🔧 Fix Cloud-native formats Performance & scale Cloud-Native GIS · step 14 of 14 Updated

dask-geopandas Uses More Memory or Time Than Plain GeoPandas

Problem statement

You reach for dask-geopandas to make a job faster and it gets slower, or it uses more memory than the single-threaded version it replaced.

This is the normal outcome for data that fits in memory. Measured on 950,256 building polygons computing total area:

plain geopandas        0.02 s
dask,  1 partition     0.14 s
dask,  4 partitions    0.13 s
dask,  8 partitions    0.13 s
dask, 16 partitions    0.18 s

Plain GeoPandas is 7.6 times faster, and more partitions make it worse. Nothing is misconfigured — the coordination costs more than the work.

Quick answer

Check four things before assuming a bug:

print(f"rows      {len(gdf):,}")
print(f"memory    {gdf.memory_usage(deep=True).sum() / 1e9:.2f} GB")
print(f"partitions {ddf.npartitions}")
print(f"tasks     {len(ddf.dask):,}")
symptom likely cause
slower, small data overhead exceeds the work — use GeoPandas
slower with more partitions too many tasks
slow spatial join no spatial_shuffle()
high memory .compute() materialising everything
slow with idle CPU tiny partitions, huge graph
Five causes of dask being slower: small data, too many partitions, missing spatial shuffle, computing everything, and repeated computes.
Only one of these is about the data being large. The rest are configuration.

Step-by-step solution

1. Cause one: the data fits in memory

The measured penalty is 7.6×, and it is not recoverable by tuning. Dask's per-task overhead is roughly constant, and computing polygon areas is about 20 ms of real work across a million rows.

The rule of thumb: below a few gigabytes on a machine with room to spare, use GeoPandas. Dask earns its overhead when the alternative is not running at all.

2. Cause two: too many partitions 

 4 partitions   0.13 s
 8 partitions   0.13 s
16 partitions   0.18 s

Each partition adds tasks, and each task adds scheduling. Once every core is busy, more partitions only add coordination.

Size partitions by memory — roughly 100 MB to 1 GB — rather than by core count.

3. Cause three: no spatial shuffle before a spatial operation

Dask partitions by row order, so rows adjacent in the file may be anywhere on the map. A spatial join then cannot exclude any partition pair and compares all of them.

left = left.spatial_shuffle()
joined = dgpd.sjoin(left, right)

The shuffle is itself expensive — a full data movement — so it pays on large joins and costs on small ones.

4. Cause four: computing more than once 

a = ddf.area.sum().compute()          # reads everything
b = ddf.length.sum().compute()        # reads everything again

Each .compute() executes the graph from the start. Either build one expression, or .persist() the shared part:

ddf = ddf.persist()                   # in memory, once
a, b = dask.compute(ddf.area.sum(), ddf.length.sum())

5. Cause five: memory rising rather than staying flat

.compute() brings the whole result into the client process. If the result is the size of the input, dask has bought nothing and cost peak memory during assembly.

Reduce before computing — an aggregate, a filter, a groupby — or write partition-wise with to_parquet and never materialise it.

Two separate compute calls each re-reading the data, against persist keeping the intermediate in memory for both.
Each `.compute()` starts from the source. `persist` is what stops the second pass.

Code examples

Example 1 — a benchmark that answers the question 

import time
import dask
import dask_geopandas as dgpd


def benchmark(gdf, operation, partitions=(1, 2, 4, 8, 16)):
    """Plain against dask at several partition counts."""
    memory_gb = gdf.memory_usage(deep=True).sum() / 1e9
    print(f"  {len(gdf):,} rows, {memory_gb:.2f} GB")

    start = time.time()
    expected = operation(gdf)
    plain = time.time() - start
    print(f"  plain geopandas      {plain:8.3f} s")

    best = None
    for n in partitions:
        ddf = dgpd.from_geopandas(gdf, npartitions=n)
        start = time.time()
        result = operation(ddf)
        result = result.compute() if hasattr(result, "compute") else result
        elapsed = time.time() - start
        tasks = len(ddf.dask)
        print(f"  dask {n:3d} partitions {elapsed:8.3f} s  "
              f"({elapsed / plain:5.1f}x)  {tasks:,} tasks")
        if best is None or elapsed < best[1]:
            best = (n, elapsed)

    if best[1] > plain:
        print(f"  -> plain geopandas wins by {best[1] / plain:.1f}x; "
              "dask only helps beyond memory")
    else:
        print(f"  -> dask wins at {best[0]} partitions "
              f"({plain / best[1]:.1f}x faster)")
    return best

Running this once on a representative sample settles the question. It also catches the case where dask wins at 4 partitions and loses at 16, which is easy to miss.

Example 2 — diagnosing a graph that is too large 

def graph_report(ddf):
    """Is the time going to work or to bookkeeping?"""
    tasks = len(ddf.dask)
    partitions = ddf.npartitions
    print(f"  {partitions} partitions, {tasks:,} tasks "
          f"({tasks / max(partitions, 1):.0f} per partition)")

    try:
        sizes = ddf.map_partitions(
            lambda part: part.memory_usage(deep=True).sum()).compute()
        print(f"  partition memory: min {sizes.min() / 1e6:.0f} MB, "
              f"median {sizes.median() / 1e6:.0f} MB, "
              f"max {sizes.max() / 1e6:.0f} MB")
        if sizes.median() < 50e6:
            print("  ! partitions under 50 MB — overhead will dominate")
        if sizes.max() > 20 * max(sizes.min(), 1):
            print("  ! very uneven partitions — one worker will finish last")
    except Exception as exc:
        print(f"  could not measure partitions: {type(exc).__name__}")

    if tasks > 100_000:
        print("  ! over 100k tasks — the scheduler is likely the bottleneck")
    return tasks

Uneven partitions are the quieter problem. Eight partitions of which one holds 80% of the rows runs at the speed of one core, and the dashboard shows seven idle workers.

Example 3 — the patterns that actually help 

import dask
import dask_geopandas as dgpd


def efficient_pipeline(path, npartitions=8):
    """Read partitioned, stay lazy, reduce before computing."""
    # 1. read a partitioned source, so reading is parallel from the start
    ddf = dgpd.read_parquet(path, npartitions=npartitions)

    # 2. filter early, so later steps carry less
    ddf = ddf[ddf["building"].notna()]

    # 3. build the whole expression before computing anything
    ddf = ddf.assign(area=ddf.geometry.area)
    by_type = ddf.groupby("building")["area"].sum()
    counts = ddf.groupby("building").size()

    # 4. compute both together, so the data is read once
    areas, sizes = dask.compute(by_type, counts)

    print(f"  {len(areas)} building types, "
          f"{sizes.sum():,} features, {areas.sum() / 1e6:.1f} km²")
    return areas, sizes

dask.compute(a, b) computes both from one traversal. Calling .compute() on each separately reads the data twice, which is the most common way a dask pipeline ends up slower than the serial version it replaced.

Explanation

Why the overhead is constant per task

For each task, dask serialises a description, sends it to a worker, deserialises it, runs it, serialises the result and sends it back. That is milliseconds regardless of whether the task takes a microsecond or a minute.

Computing areas for a million polygons is roughly 20 ms of NumPy and GEOS work. Split into 16 tasks, the coordination is comparable to the work — measured, 0.18 s against 0.02 s.

The break-even is where per-task work exceeds per-task overhead. For cheap row-wise operations that means very large partitions; for expensive ones — overlays, complex predicates, per-row Python — it arrives much sooner.

Why memory can go up

Three mechanisms.

.compute() assembles the whole result in the client process, and during assembly both the partition results and the concatenated output exist.

Workers hold several partitions at once — one being processed, others queued — so peak worker memory is a multiple of the partition size.

And a shuffle materialises intermediate data across partitions, which for a spatial shuffle is the whole dataset rearranged.

The fix is to reduce before computing, or to write partition-wise and never assemble.

Why the dashboard is worth using

dask.distributed's dashboard shows where time goes: task execution, serialisation, scheduling, or waiting.

A profile dominated by scheduling means too many small tasks. One dominated by serialisation means partitions are being moved when they should not be. One dominated by a single long bar means uneven partitions.

None of those is visible from a wall-clock number, and each has a different fix.

Why the right question is "does this fit in memory"

Dask's value is running computations that otherwise cannot run. It is not a general speed-up.

If the data fits comfortably, the honest comparison is against GeoPandas with the obvious optimisations — vectorised operations rather than .apply, a spatial index, the right dtype. Those usually beat dask by more than dask's parallelism could ever recover.

If it does not fit, dask is competing against not running at all, and a 7.6× overhead on work you could not otherwise do is an excellent trade.

Partition sizing with under 50 MB dominated by overhead, 100 MB to 1 GB as the target, and over 4 GB starving memory.
Sizing by core count optimises for parallelism and ignores memory, which is usually the binding constraint.

Edge cases or notes

  • Below a few gigabytes, plain GeoPandas wins. Measured 7.6× here.
  • More partitions is not better. 16 was slower than 8.
  • spatial_shuffle() before any spatial join, and know it costs a full data movement.
  • dask.compute(a, b) computes both from one pass.
  • .persist() for an intermediate reused several ways.
  • Reduce before computing, or the client assembles the whole result.
  • Uneven partitions run at the speed of the largest.
  • Fix the GeoPandas code first — .apply over rows is usually the real problem.

FAQ

Why is dask-geopandas slower than GeoPandas?

Because coordination costs more than the work when the data fits in memory. Measured on 950,256 polygons, plain GeoPandas was 7.6× faster.

Should more partitions make it faster?

Only up to the point where every core is busy. Beyond that, extra partitions add scheduling — 16 was slower than 8 here.

Why is my dask spatial join so slow?

No spatial shuffle. Partitions are arbitrary row ranges by default, so the join must compare every partition pair.

Why does dask use more memory than GeoPandas?

.compute() assembles the whole result in the client, workers hold several partitions at once, and shuffles materialise intermediates. Reduce before computing.

How big should partitions be?

100 MB to 1 GB. Sizing by core count optimises for parallelism and ignores memory, which is usually the binding constraint.

When is dask-geopandas worth it?

When the data exceeds memory, when the per-row work is expensive, or when the input is already partitioned on disk.

What should I try before dask?

Vectorise the GeoPandas code, use the spatial index, pick better dtypes, and read only the columns you need. Those usually beat anything parallelism can recover.