dask-geopandas Uses More Memory or Time Than Plain GeoPandas

Problem statement

You reach for dask-geopandas to make a job faster and it gets slower, or it uses more memory than the single-threaded version it replaced.

This is the normal outcome for cheap work on data that fits in memory. Measured on 954,489 Overture building footprints in Rotterdam and Den Haag, computing total area (Example 1):

plain geopandas        0.037 s
dask,  1 partition     0.108 s
dask,  4 partitions    0.112 s
dask,  8 partitions    0.107 s
dask, 16 partitions    0.144 s
dask, 64 partitions    0.215 s

Plain GeoPandas is 2.9 times faster than dask's best, and more partitions make it worse. Nothing is misconfigured — the coordination costs more than the work. Buffer each polygon by 50 m before taking its area and the order flips: 12.7 s plain, 1.5 s in dask at 16 partitions.

Quick answer

Check four things before assuming a bug:

import dask_geopandas as dgpd
import shapely

ddf = dgpd.read_parquet("buildings")
coords_gb = shapely.get_num_coordinates(gdf.geometry.values).sum() * 16 / 1e9
print(f"rows       {len(gdf):,}")
print(f"memory     {gdf.memory_usage(deep=True).sum() / 1e9:.2f} GB + {coords_gb:.2f} GB of coordinates")
print(f"partitions {ddf.npartitions}")
print(f"tasks      {len(ddf.dask):,}")

rows       954,489
memory     0.08 GB + 0.11 GB of coordinates
partitions 8
tasks      16

memory_usage(deep=True) does not look inside the geometries, so count their coordinates as well.

Data that fits, too many partitions, no shuffle, repeated compute

1. Cause one: the data fits in memory

The measured penalty is 2.9× at best, and it is not recoverable by tuning. Dask's per-task overhead is roughly constant, and computing polygon areas is 0.037 s of real work across a million rows.

The rule of thumb: below a few gigabytes on a machine with room to spare, use GeoPandas. Dask earns its overhead when the alternative is not running at all.

2. Cause two: too many partitions

 4 partitions   0.112 s
 8 partitions   0.107 s
16 partitions   0.144 s
64 partitions   0.215 s

Each partition adds tasks, and each task adds scheduling. Once every core is busy, more partitions only add coordination.

Size partitions by memory — roughly 100 MB to 1 GB — rather than by core count.

3. Cause three: no spatial shuffle before a spatial operation

Dask partitions by row order, so rows adjacent in the file may be anywhere on the map. A spatial join then cannot exclude any partition pair and compares all of them.

import geopandas as gpd

left = dgpd.read_parquet("buildings").spatial_shuffle()
right = dgpd.from_geopandas(gpd.read_parquet("localities.parquet"), npartitions=1)
joined = dgpd.sjoin(left, right)
print(f"{joined.npartitions} partition pairs; {len(joined):,} building-locality matches")

8 partition pairs; 954,799 building-locality matches

The right side is read with GeoPandas on purpose: under pandas 3, joining two frames that both came from dgpd.read_parquet fails with ValueError: The truth value of a Series is ambiguous, because each carries its spatial partitions in .attrs and the join compares them with ==.

The shuffle is itself expensive — a full data movement — so it pays on large joins and costs on small ones.

4. Cause four: computing more than once

import time

import dask

start = time.perf_counter()
a = ddf.area.sum().compute()          # reads everything
b = ddf.length.sum().compute()        # reads everything again
print(f"two computes        {time.perf_counter() - start:.2f} s")

two computes        1.22 s

Each .compute() executes the graph from the start — reading the 8 files twice here. Either build one expression, or .persist() the shared part:

start = time.perf_counter()
a, b = dask.compute(ddf.area.sum(), ddf.length.sum())
print(f"one dask.compute    {time.perf_counter() - start:.2f} s")

start = time.perf_counter()
kept = ddf.persist()                  # in memory, once
print(f"persist             {time.perf_counter() - start:.2f} s")
start = time.perf_counter()
a, b = dask.compute(kept.area.sum(), kept.length.sum())
print(f"after persist       {time.perf_counter() - start:.2f} s")

one dask.compute    0.56 s
persist             0.49 s
after persist       0.04 s

One dask.compute did both sums in about half the time of two separate computes, because the files were read once. persist costs the read up front and makes everything after it nearly free — worth it when the same intermediate feeds several results.

5. Cause five: memory rising rather than staying flat

.compute() brings the whole result into the client process. If the result is the size of the input, dask has bought nothing and cost peak memory during assembly.

Reduce before computing — an aggregate, a filter, a groupby — or write partition-wise with to_parquet and never materialise it.

A benchmark and a graph-size diagnosis

The data used on this page

The buildings are Overture Maps' building footprints for Rotterdam, Delft and Den Haag — every footprint whose bounding box touches 4.2–4.6° E, 51.85–52.15° N, about 950,000 of them — read straight from Overture's public S3 bucket with DuckDB, projected to the Dutch RD New grid (EPSG:28992) so areas are in square metres, and saved twice: as one buildings.parquet file and as a buildings/ directory of 8 files written in Hilbert-curve order. The 65 towns and cities (Overture locality division areas) in the same box are used for the join. Overture keeps only recent releases, so the setup looks up the newest one and the counts can move a little between releases.

import os
import re

import dask_geopandas as dgpd
import duckdb
import geopandas as gpd
import requests

BBOX = (4.2, 51.85, 4.6, 52.15)          # Rotterdam, Delft and Den Haag


def latest_overture_release():
    listing = requests.get("https://overturemaps-us-west-2.s3.amazonaws.com/",
                           params={"list-type": "2", "prefix": "release/", "delimiter": "/"},
                           timeout=60).text
    return sorted(re.findall(r"<Prefix>release/([^<]+)/</Prefix>", listing))[-1]


def overture(theme, kind, where, path):
    """Copy the features inside BBOX from Overture's public bucket to a local GeoParquet file."""
    if os.path.exists(path):
        return
    con = duckdb.connect()
    con.execute("install httpfs; load httpfs; install spatial; load spatial;")
    con.execute("set s3_region = 'us-west-2'; set enable_progress_bar = false")
    x0, y0, x1, y1 = BBOX
    release = latest_overture_release()
    con.execute(f"""
        copy (select * exclude (bbox) from read_parquet(
                's3://overturemaps-us-west-2/release/{release}/theme={theme}/type={kind}/*.parquet',
                hive_partitioning = 0)
              where bbox.xmin < {x1} and bbox.xmax > {x0} and bbox.ymin < {y1} and bbox.ymax > {y0}
                and {where})
        to '{path}' (format parquet)""")
    with open(path + ".release", "w") as note:
        note.write(release)


overture("buildings", "building", "true",
         "overture_buildings.parquet")
overture("divisions", "division_area", "subtype = 'locality' and country = 'NL'", "overture_localities.parquet")

if not os.path.exists("buildings.parquet"):
    raw = gpd.read_parquet("overture_buildings.parquet", columns=["id", "subtype", "class", "height", "geometry"])
    raw.to_crs(28992).to_parquet("buildings.parquet")          # RD New, metres
if not os.path.exists("localities.parquet"):
    places = gpd.read_parquet("overture_localities.parquet")
    places["name"] = places["names"].map(lambda n: n["primary"])
    places[["id", "name", "geometry"]].to_crs(28992).to_parquet("localities.parquet")
if not os.path.exists("buildings"):
    ordered = gpd.read_parquet("buildings.parquet")
    ordered = ordered.iloc[ordered.geometry.hilbert_distance().argsort()]
    dgpd.from_geopandas(ordered.reset_index(drop=True), npartitions=8).to_parquet("buildings")

gdf = gpd.read_parquet("buildings.parquet")
print(f"Overture release {open('overture_buildings.parquet.release').read()}: "
      f"{len(gdf):,} buildings, {len(gpd.read_parquet('localities.parquet'))} localities; "
      f"buildings/ holds {len(os.listdir('buildings'))} files")

Overture release 2026-09-23.1: 954,489 buildings, 65 localities; buildings/ holds 8 files

Example 1 — a benchmark that answers the question

import time
import dask
import dask_geopandas as dgpd
import shapely


def frame_bytes(frame):
    """Columns plus coordinates: memory_usage(deep=True) does not see inside geometries."""
    return (frame.memory_usage(deep=True).sum()
            + shapely.get_num_coordinates(frame.geometry.values).sum() * 16)


def benchmark(gdf, operation, partitions=(1, 2, 4, 8, 16)):
    """Plain against dask at several partition counts."""
    print(f"  {len(gdf):,} rows, {frame_bytes(gdf) / 1e9:.2f} GB")

    start = time.perf_counter()
    operation(gdf)
    plain = time.perf_counter() - start
    print(f"  plain geopandas      {plain:8.3f} s")

    best = None
    for n in partitions:
        ddf = dgpd.from_geopandas(gdf, npartitions=n)
        start = time.perf_counter()
        result = operation(ddf)
        result = result.compute() if hasattr(result, "compute") else result
        elapsed = time.perf_counter() - start
        tasks = len(ddf.dask)
        print(f"  dask {n:3d} partitions {elapsed:8.3f} s  "
              f"({elapsed / plain:5.1f}x)  {tasks:,} tasks")
        if best is None or elapsed < best[1]:
            best = (n, elapsed)

    if best[1] > plain:
        print(f"  -> plain geopandas wins by {best[1] / plain:.1f}x; "
              "dask only helps beyond memory")
    else:
        print(f"  -> dask wins at {best[0]} partitions "
              f"({plain / best[1]:.1f}x faster)")
    return best

Summing the areas, then buffering each building by 50 m first:

print("area")
benchmark(gdf, lambda frame: frame.geometry.area.sum(), partitions=(1, 2, 4, 8, 16, 64))
print("buffer 50 m")
benchmark(gdf, lambda frame: frame.geometry.buffer(50).area.sum(), partitions=(1, 2, 4, 8, 16, 64))

area
  954,489 rows, 0.20 GB
  plain geopandas         0.037 s
  dask   1 partitions    0.108 s  (  2.9x)  1 tasks
  dask   2 partitions    0.106 s  (  2.9x)  2 tasks
  dask   4 partitions    0.112 s  (  3.0x)  4 tasks
  dask   8 partitions    0.107 s  (  2.9x)  8 tasks
  dask  16 partitions    0.144 s  (  3.9x)  16 tasks
  dask  64 partitions    0.215 s  (  5.8x)  64 tasks
  -> plain geopandas wins by 2.9x; dask only helps beyond memory
buffer 50 m
  954,489 rows, 0.20 GB
  plain geopandas        12.746 s
  dask   1 partitions   13.146 s  (  1.0x)  1 tasks
  dask   2 partitions    6.996 s  (  0.5x)  2 tasks
  dask   4 partitions    3.959 s  (  0.3x)  4 tasks
  dask   8 partitions    2.135 s  (  0.2x)  8 tasks
  dask  16 partitions    1.495 s  (  0.1x)  16 tasks
  dask  64 partitions    1.431 s  (  0.1x)  64 tasks
  -> dask wins at 64 partitions (8.9x faster)

Running this once on a representative sample settles the question. It also catches the case where dask wins at 4 partitions and loses at 16, which is easy to miss.

Example 2 — diagnosing a graph that is too large

def graph_report(ddf):
    """Is the time going to work or to bookkeeping?"""
    tasks = len(ddf.dask)
    partitions = ddf.npartitions
    print(f"  {partitions} partitions, {tasks:,} tasks "
          f"({tasks / max(partitions, 1):.0f} per partition)")

    try:
        sizes = ddf.map_partitions(frame_bytes, meta=("bytes", "int64")).compute()
        print(f"  partition memory: min {sizes.min() / 1e6:.2f} MB, "
              f"median {sizes.median() / 1e6:.2f} MB, "
              f"max {sizes.max() / 1e6:.2f} MB")
        if sizes.median() < 50e6:
            print("  ! partitions under 50 MB — overhead will dominate")
        if sizes.max() > 20 * max(sizes.min(), 1):
            print("  ! very uneven partitions — one worker will finish last")
    except Exception as exc:
        print(f"  could not measure partitions: {type(exc).__name__}")

    if tasks > 100_000:
        print("  ! over 100k tasks — the scheduler is likely the bottleneck")
    return tasks

On the 8 files as written, and on the same data cut into 2,000 partitions:

graph_report(dgpd.read_parquet("buildings"))
graph_report(dgpd.from_geopandas(gdf, npartitions=2000))

  8 partitions, 16 tasks (2 per partition)
  partition memory: min 24.19 MB, median 25.31 MB, max 28.38 MB
  ! partitions under 50 MB — overhead will dominate
  2000 partitions, 2,000 tasks (1 per partition)
  partition memory: min 0.08 MB, median 0.10 MB, max 0.20 MB
  ! partitions under 50 MB — overhead will dominate

The first warning deserves a second look: the 8 files are 25 MB each, and those same partitions buffered six times faster than plain GeoPandas in Example 1. Small partitions make cheap work overhead-bound; heavy work absorbs the overhead. The 2,000-partition graph is the real problem — 0.1 MB each, and a task for every one.

Uneven partitions are the quieter problem. Eight partitions of which one holds 80% of the rows runs at the speed of one core, and the dashboard shows seven idle workers.

Example 3 — the patterns that actually help

import dask
import dask_geopandas as dgpd


def efficient_pipeline(path):
    """Read partitioned, stay lazy, reduce before computing."""
    # 1. read a partitioned source, so reading is parallel from the start
    ddf = dgpd.read_parquet(path)

    # 2. filter early, so later steps carry less
    ddf = ddf[ddf["class"].notnull()]            # dask Series has notnull(), not notna()

    # 3. build the whole expression before computing anything
    ddf = ddf.assign(area=ddf.geometry.area)
    by_type = ddf.groupby("class")["area"].sum()
    counts = ddf.groupby("class").size()

    # 4. compute both together, so the data is read once
    areas, sizes = dask.compute(by_type, counts)

    print(f"  {len(areas)} building classes, "
          f"{sizes.sum():,} features, {areas.sum() / 1e6:.1f} km²")
    return areas, sizes

areas, sizes = efficient_pipeline("buildings")

  65 building classes, 633,827 features, 98.2 km²

dask.compute(a, b) computes both from one traversal. Calling .compute() on each separately reads the data twice, which is the most common way a dask pipeline ends up slower than the serial version it replaced.

Constant per-task overhead and rising memory

Why the overhead is constant per task

For each task, dask serialises a description, sends it to a worker, deserialises it, runs it, serialises the result and sends it back. That is milliseconds regardless of whether the task takes a microsecond or a minute.

Computing areas for 954,489 polygons is 0.037 s of NumPy and GEOS work. Split into 16 tasks, the coordination outweighs the work — measured, 0.144 s against 0.037 s.

The break-even is where per-task work exceeds per-task overhead. For cheap row-wise operations that means very large partitions; for expensive ones — overlays, complex predicates, per-row Python — it arrives much sooner.

Why memory can go up

Three mechanisms.

.compute() assembles the whole result in the client process, and during assembly both the partition results and the concatenated output exist.

Workers hold several partitions at once — one being processed, others queued — so peak worker memory is a multiple of the partition size.

And a shuffle materialises intermediate data across partitions, which for a spatial shuffle is the whole dataset rearranged.

The fix is to reduce before computing, or to write partition-wise and never assemble.

Why the dashboard is worth using

dask.distributed's dashboard shows where time goes: task execution, serialisation, scheduling, or waiting.

A profile dominated by scheduling means too many small tasks. One dominated by serialisation means partitions are being moved when they should not be. One dominated by a single long bar means uneven partitions.

None of those is visible from a wall-clock number, and each has a different fix.

Why the right question is "does this fit in memory"

Dask's value is running computations that otherwise cannot run. It is not a general speed-up.

If the data fits comfortably, the honest comparison is against GeoPandas with the obvious optimisations — vectorised operations rather than .apply, a spatial index, the right dtype. Those usually beat dask by more than dask's parallelism could ever recover.

If it does not fit, dask is competing against not running at all, and a 2.9× overhead on work you could not otherwise do is an excellent trade.

Fits-in-memory work, uneven partitions and reducing first

• For cheap work on data that fits, plain GeoPandas wins. Measured 2.9× here on an area sum; a 50 m buffer went the other way.
• More partitions is not better. 64 summed areas in 0.215 s against 0.107 s for 8.
• spatial_shuffle() before any spatial join, and know it costs a full data movement.
• dask.compute(a, b) computes both from one pass.
• .persist() for an intermediate reused several ways.
• Reduce before computing, or the client assembles the whole result.
• Uneven partitions run at the speed of the largest.
• Fix the GeoPandas code first — .apply over rows is usually the real problem.

Internal links

• How to scale a GeoPandas job with dask-geopandas — when it does help
• Why GeoPandas is slow: the four real bottlenecks — check these before parallelising
• GeoPandas .apply() takes hours: how to fix it — the most common real cause
• Threads, processes and the GIL: parallelism in Python GIS explained — the alternatives
• How to write partitioned GeoParquet and query it with filters — the input layout that helps
• Lazy loading explained: why xarray reads nothing until you ask — the same compute-once rule
• How to replace a row loop with vectorised GeoPandas operations — the cheaper speed-up
• How to profile a slow Python GIS script and find the real bottleneck — before choosing a tool

FAQ

Why is dask-geopandas slower than GeoPandas?

Because coordination costs more than the work when the work is cheap and the data fits in memory. Measured on 954,489 polygons, plain GeoPandas summed their areas 2.9× faster than dask's best — and buffering them first, dask at 16 partitions was 8.5× faster than plain.

Should more partitions make it faster?

Only up to the point where every core is busy. Beyond that, extra partitions add scheduling — 64 was twice as slow as 8 for the area sum here.

Why is my dask spatial join so slow?

No spatial shuffle. Partitions are arbitrary row ranges by default, so the join must compare every partition pair.

Why does dask use more memory than GeoPandas?

.compute() assembles the whole result in the client, workers hold several partitions at once, and shuffles materialise intermediates. Reduce before computing.

How big should partitions be?

100 MB to 1 GB. Sizing by core count optimises for parallelism and ignores memory, which is usually the binding constraint.

When is dask-geopandas worth it?

When the data exceeds memory, when the per-row work is expensive, or when the input is already partitioned on disk.

What should I try before dask?

Vectorise the GeoPandas code, use the spatial index, pick better dtypes, and read only the columns you need. Those usually beat anything parallelism can recover.