How to Scale a GeoPandas Job with dask-geopandas
Problem statement
dask-geopandas partitions a GeoDataFrame and runs operations across partitions, in parallel or out of core. It is the right tool for data that does not fit in memory, and it is slower than plain GeoPandas for data that does.
Measured on 950,256 building polygons computing the total area:
dask, 1 partition 0.14 s
dask, 4 partitions 0.13 s
dask, 8 partitions 0.13 s
dask, 16 partitions 0.18 s
plain geopandas 0.02 s
Plain GeoPandas is 7.6 times faster. The parallelism is real; the coordination overhead is larger than the work.
Knowing when that inverts is the whole skill.
Quick answer
Use dask-geopandas when at least one is true:
the data does not fit in memory
the per-row work is heavy (buffering, overlay, complex predicates)
the data is already partitioned on disk
the operation is embarrassingly parallel and takes minutes
Use plain GeoPandas otherwise:
import dask_geopandas as dgpd
ddf = dgpd.read_parquet("buildings/", npartitions=8)
ddf = ddf.spatial_shuffle() # co-locate nearby rows
result = ddf.geometry.buffer(50).area.sum().compute()
Step-by-step solution
1. Establish whether you have a scale problem
print(f"{len(gdf):,} rows, {gdf.memory_usage(deep=True).sum() / 1e9:.2f} GB")Under a few gigabytes on a machine with plenty of memory, plain GeoPandas is almost always faster. The measured 7.6× penalty is the overhead you pay for nothing.
2. Size the partitions by memory, not by core count
A partition should be big enough that the work dwarfs the task overhead, and small enough that several fit in memory at once.
Around 100 MB to 1 GB per partition is a reasonable range. npartitions = cores is a common default and usually wrong — it optimises for parallelism and ignores memory.
3. Spatially shuffle before any spatial operation
ddf = ddf.spatial_shuffle()By default, partitions are arbitrary row ranges, so a spatial join must compare every left partition with every right one. After a spatial shuffle, rows are grouped by location, and only overlapping partitions need comparing.
For spatial joins and overlays this is the difference between quadratic and near-linear.
4. Keep the computation lazy until the end
result = (ddf[ddf["building"] == "yes"]
.assign(area=lambda d: d.geometry.area)
.groupby("addr:postcode")["area"].sum()
.compute())Every .compute() executes the graph from the start. Build the whole expression, compute once.
5. Prefer partitioned Parquet as the input
dgpd.read_parquet on a partitioned directory reads each file as a partition, in parallel, with no shuffle needed. Reading a single GeoPackage and repartitioning does the expensive part serially before the parallelism starts.
Code examples
Example 1 — deciding whether dask is worth it
import time
import geopandas as gpd
import dask_geopandas as dgpd
def should_i_use_dask(gdf, operation, npartitions=8, sample=50_000):
"""Time the operation both ways on a sample and extrapolate."""
memory_gb = gdf.memory_usage(deep=True).sum() / 1e9
print(f" {len(gdf):,} rows, {memory_gb:.2f} GB in memory")
subset = gdf.sample(min(sample, len(gdf)), random_state=0)
start = time.time()
operation(subset)
plain = time.time() - start
ddf = dgpd.from_geopandas(subset, npartitions=npartitions)
start = time.time()
operation(ddf).compute() if hasattr(operation(ddf), "compute") \
else operation(ddf)
parallel = time.time() - start
print(f" on {len(subset):,} rows: plain {plain:.2f}s, "
f"dask {parallel:.2f}s ({parallel / plain:.1f}x)")
if memory_gb > 4:
print(" -> use dask: the data is large enough that memory decides")
elif parallel < plain:
print(" -> use dask: the work per row is heavy enough to parallelise")
else:
print(" -> use plain geopandas: dask overhead exceeds the work")
return parallel < plain or memory_gb > 4Running the comparison on a sample takes seconds and settles the question with a number rather than an assumption.
Example 2 — a spatial join that scales
import dask_geopandas as dgpd
def scaled_spatial_join(left_path, right_path, npartitions=16,
predicate="intersects"):
"""Partitioned join with the shuffle that makes it tractable."""
left = dgpd.read_parquet(left_path, npartitions=npartitions)
right = dgpd.read_parquet(right_path)
print(f" left {left.npartitions} partitions, "
f"right {right.npartitions} partitions")
left = left.spatial_shuffle()
print(" spatially shuffled: partitions now group nearby geometry")
joined = dgpd.sjoin(left, right, predicate=predicate)
print(f" join graph built ({len(joined.dask):,} tasks)")
return joinedWithout spatial_shuffle, a join between 16 and 16 partitions considers 256 pairs. With it, only the pairs whose bounding boxes overlap — typically a handful per partition.
The shuffle itself is expensive: it is a full data movement. It pays when the join is large and does not when it is small, which is another instance of the same rule.
Example 3 — writing partitioned output
import dask_geopandas as dgpd
def process_and_write(input_path, output_path, npartitions=16,
buffer_m=50):
"""Read partitions, transform, write partitions — nothing in memory."""
ddf = dgpd.read_parquet(input_path, npartitions=npartitions)
ddf = ddf.to_crs(ddf.crs) # no-op, but forces the check
ddf["geometry"] = ddf.geometry.buffer(buffer_m)
ddf["area_m2"] = ddf.geometry.area
ddf.to_parquet(output_path, write_index=False)
print(f" wrote {ddf.npartitions} partitions to {output_path}")
return output_pathPartitioned in, partitioned out, and nothing ever holds the whole dataset. That is the case dask-geopandas is genuinely for: a dataset larger than memory, transformed row-wise.
Note that to_crs on a dask GeoDataFrame reprojects per partition, which is correct — reprojection is row-wise. Operations that are not row-wise, such as a dissolve across the whole dataset, need a shuffle and are much more expensive.
Explanation
Why dask is slower on small data
Every dask operation builds a task graph, serialises the tasks, schedules them, and collects the results. That costs milliseconds per task regardless of how much work the task does.
Computing the area of 950,256 polygons is about 20 ms of actual work. Split into 16 partitions, the scheduling overhead alone exceeds it — hence 0.18 s against 0.02 s.
The crossover is where per-partition work exceeds per-task overhead. For cheap row-wise operations that needs very large partitions; for expensive ones — an overlay, a complex predicate — it arrives much sooner.
Why more partitions is not better
The measurement shows 16 partitions slower than 8, which is the overhead curve turning up.
Each partition adds tasks, and each task adds scheduling. Beyond the point where every core is busy, more partitions only add coordination.
Size partitions by memory: large enough that the work is substantial, small enough that several fit at once. Then check that the count is at least the number of cores, and stop.
Why the spatial shuffle matters
Dask partitions a GeoDataFrame by row order. Rows adjacent in the file may be anywhere on the map.
A spatial join then cannot rule out any partition pair: any left partition might contain a row that intersects any right partition. So the join degenerates to comparing all pairs.
spatial_shuffle reorders rows by a space-filling curve and repartitions, so each partition covers a compact region with a small bounding box. The join can then skip most pairs on bounding boxes alone.
Why partitioned Parquet is the right input
Reading a single large file is serial: one process decodes it, then repartitions in memory. The parallelism starts after the slowest part.
A partitioned directory gives each worker a file. Reading is parallel from the start, and if the partitions were written spatially sorted, the shuffle is unnecessary too.
That makes the storage layout part of the compute design, which is the recurring theme of every cloud-native format.
Edge cases or notes
- Plain GeoPandas is faster below a few gigabytes — 7.6× on the measured workload.
- Size partitions by memory, roughly 100 MB to 1 GB.
- More partitions is not better. 16 was slower than 8 here.
spatial_shufflebefore any spatial join or overlay.- One
.compute()at the end, not per step. - Partitioned Parquet input makes reading parallel from the start.
- Row-wise operations parallelise well. Dissolve, sort and global joins need shuffles.
- The dask scheduler dashboard shows whether time goes to tasks or to coordination.
Internal links
- How to write partitioned GeoParquet and query it with filters — the input layout
- dask-geopandas uses more memory or time than plain GeoPandas — when it goes wrong
- Threads, processes and the GIL: parallelism in Python GIS explained — the alternatives
- Why GeoPandas is slow: the four real bottlenecks — check these first
- How to speed up batch GIS jobs with parallel processing in Python — file-level parallelism
- Lazy loading explained: why xarray reads nothing until you ask — the same laziness for arrays
- How to process a very large GeoPackage in chunks with Python — manual chunking
- Spatial indexes explained: R-trees and why spatial joins are fast — what the shuffle enables
FAQ
Is dask-geopandas faster than GeoPandas?
Not usually. On 950,256 polygons computing area, plain GeoPandas was 7.6× faster. Dask pays off when the data exceeds memory or the per-row work is heavy.
How many partitions should I use?
Enough that each holds 100 MB to 1 GB. Sizing by core count is a common default and usually wrong.
Why is my spatial join so slow?
Probably no spatial shuffle. Without it, partitions are arbitrary row ranges and the join must compare every pair.
When should I use dask-geopandas?
When the data does not fit in memory, when the per-row work is expensive, or when the data is already partitioned on disk.
Does to_crs work on a dask GeoDataFrame?
Yes — reprojection is row-wise, so it parallelises cleanly per partition.
Why did adding partitions make it slower?
Beyond the point where every core is busy, extra partitions only add scheduling overhead. 16 was slower than 8 in the measurement here.
What input format should I use?
Partitioned GeoParquet. Reading a single large file is serial, so the parallelism starts only after the slowest step.