How to Write Partitioned GeoParquet and Query It with Filters
Problem statement
GeoParquet is dramatically smaller and faster than the alternatives. Measured on 59,391 real OpenStreetMap building polygons:
GeoParquet 5.06 MB read 0.06 s
GeoPackage 16.59 MB read 0.20 s
GeoJSON 27.04 MB read 0.49 s
Shapefile (all parts) 50.31 MB read 0.16 s
Ten times smaller than a shapefile and eight times faster to read than GeoJSON. But two features people expect from it need explicit setup, and without that setup they silently do nothing:
- Spatial filtering on read needs a covering-bbox column, or
geopandasrefuses. - Row-group skipping needs more than one row group, and the default is often one.
Quick answer
import geopandas as gpd
# spatial filtering needs a bbox column written in
gdf.to_parquet("buildings.parquet", write_covering_bbox=True)
subset = gpd.read_parquet("buildings.parquet",
bbox=(-2.25, 53.47, -2.23, 53.49))Without it:
ValueError: Specifying 'bbox' not supported for this Parquet file
(it should either have a bbox covering column or use
'point' encoding).
The covering-bbox column cost 44% more file size here — 7.30 MB against 5.06 MB — and that is the price of the feature.
Step-by-step solution
1. Check how many row groups you actually have
import pyarrow.parquet as pq
f = pq.ParquetFile("buildings.parquet")
print(f"{f.metadata.num_row_groups} row groups, {f.metadata.num_rows:,} rows")1 row groups, 950,256 rows
One row group. Predicate pushdown works by reading each row group's column statistics and skipping groups that cannot match — with one group there is nothing to skip, and every filter reads the whole file.
The default row-group size is around a million rows, so most single-file GeoParquet has exactly one.
gdf.to_parquet(path, row_group_size=50_000)2. Add the covering-bbox column if you want spatial filtering
gdf.to_parquet(path, write_covering_bbox=True)This adds four float columns — xmin, ymin, xmax, ymax — per row. Their per-row-group statistics let a reader skip groups whose bounding boxes miss the query.
The cost is real: 7.30 MB against 5.06 MB, a factor of 1.44. Worth it for a file that will be queried spatially many times; not worth it for one read end to end.
3. Sort spatially before writing
This is the step that makes the statistics useful. Row-group statistics only help if the rows in a group are spatially close — otherwise every group's bounding box covers the whole dataset and none can be skipped.
gdf = gdf.sort_values("hilbert") # or geohash, or Morton order
gdf.to_parquet(path, row_group_size=50_000, write_covering_bbox=True)Without a spatial sort, adding the bbox column and shrinking the row groups buys almost nothing.
4. Partition by a directory key for very large datasets
buildings/
tile=530_-22/part.parquet
tile=530_-23/part.parquet
Hive-style partitioning puts the key in the path, so a reader can skip whole files without opening them. Measured on 950,256 rows in 24 tiles:
read one tile (142,272 rows) 0.093 s
read everything (950,256 rows) 0.617 s
Roughly proportional to rows, which is what you want — the partitioning removed the rest entirely.
Note that geopandas.to_parquet does not accept partition_cols; write the partitions yourself with a groupby, or use dask_geopandas.
5. Choose the partition key from the query
Partition by whatever you filter on most: a tile, a region, a date, a category. Partitioning by something you never filter on adds directories and no benefit.
Code examples
Example 1 — writing a file that can actually be filtered
import numpy as np
import geopandas as gpd
import pyarrow.parquet as pq
def write_queryable(gdf, path, row_group_size=50_000, sort=True):
"""Spatially sorted, sensible row groups, covering bbox."""
if sort:
bounds = gdf.geometry.bounds
cx = ((bounds["minx"] + bounds["maxx"]) / 2).to_numpy()
cy = ((bounds["miny"] + bounds["maxy"]) / 2).to_numpy()
# Morton (Z-order) interleave of scaled integer coordinates
def scale(v):
lo, hi = v.min(), v.max()
return ((v - lo) / max(hi - lo, 1e-12) * (2 ** 20 - 1)).astype(np.uint64)
def interleave(a, b):
out = np.zeros_like(a)
for bit in range(20):
out |= ((a >> bit) & 1) << (2 * bit + 1)
out |= ((b >> bit) & 1) << (2 * bit)
return out
gdf = gdf.iloc[np.argsort(interleave(scale(cx), scale(cy)))]
gdf.to_parquet(path, row_group_size=row_group_size,
write_covering_bbox=True)
meta = pq.ParquetFile(path).metadata
print(f" {meta.num_rows:,} rows in {meta.num_row_groups} row groups")
print(f" {meta.serialized_size / 1e3:.0f} kB of metadata")
if meta.num_row_groups == 1:
print(" ! one row group — no statistics-based skipping is possible")
return pathThe Morton sort is the part that makes everything else work. Twenty lines, once, and every spatial filter afterwards can skip most of the file.
Example 2 — writing Hive partitions
import os
import shutil
import numpy as np
import geopandas as gpd
def write_partitioned(gdf, out_dir, cell_deg=0.025, row_group_size=50_000):
"""One file per tile, in a directory layout a reader can prune."""
shutil.rmtree(out_dir, ignore_errors=True)
os.makedirs(out_dir)
centroids = gdf.geometry.representative_point()
tile = (np.floor(centroids.x / cell_deg).astype(int).astype(str) + "_" +
np.floor(centroids.y / cell_deg).astype(int).astype(str))
sizes = []
for key, part in gdf.groupby(tile.values):
directory = os.path.join(out_dir, f"tile={key}")
os.makedirs(directory, exist_ok=True)
path = os.path.join(directory, "part.parquet")
part.to_parquet(path, row_group_size=row_group_size,
write_covering_bbox=True)
sizes.append((key, len(part), os.path.getsize(path)))
total = sum(s for _, _, s in sizes)
print(f" {len(sizes)} partitions, {total / 1e6:.1f} MB total")
print(f" rows per partition: min {min(n for _, n, _ in sizes):,}, "
f"median {int(np.median([n for _, n, _ in sizes])):,}, "
f"max {max(n for _, n, _ in sizes):,}")
if max(n for _, n, _ in sizes) > 20 * max(1, min(n for _, n, _ in sizes)):
print(" ! very uneven partitions — consider a quadtree rather than "
"a fixed grid")
return out_dirEven partition sizes matter more than the exact scheme. A fixed grid over clustered data produces one enormous partition and many tiny ones, which is the worst of both — no pruning benefit and per-file overhead.
Example 3 — reading with the filters that exist
import geopandas as gpd
import pyarrow.dataset as ds
def read_filtered(path, bbox=None, columns=None, where=None):
"""Spatial and attribute filtering, with a clear fallback."""
try:
gdf = gpd.read_parquet(path, bbox=bbox, columns=columns,
filters=where)
print(f" {len(gdf):,} rows via pushdown")
return gdf
except ValueError as exc:
if "bbox" not in str(exc):
raise
print(f" ! {exc}")
print(" falling back to reading everything and clipping")
gdf = gpd.read_parquet(path, columns=columns, filters=where)
if bbox:
gdf = gdf.cx[bbox[0]:bbox[2], bbox[1]:bbox[3]]
print(f" {len(gdf):,} rows after clipping")
return gdfMaking the fallback loud is the point. A silent fallback turns "spatial pushdown" into "read the whole file and filter in memory", which works and does not scale, and nobody notices until the dataset grows.
Explanation
Why one row group defeats pushdown
Parquet stores min and max for each column in each row group. A reader evaluating population > 10000 reads those statistics, skips groups whose maximum is below the threshold, and decodes the rest.
With one row group there is nothing to skip. The filter still works — it is applied after decoding — but it saves no I/O.
The default row-group size of around a million rows means most single-file GeoParquet has exactly one group, and every filter reads everything. Setting row_group_size to something like 50,000 is what turns pushdown on.
Why spatial sorting is the precondition
Statistics-based skipping works when a group's range is narrow. If rows are in arbitrary order, every group's bounding box covers most of the dataset, and no query can exclude any group.
Sorting by a space-filling curve — Hilbert or Morton — puts spatially close rows next to each other, so each row group covers a compact region. Then a bounding-box query genuinely excludes most groups.
Sorting is therefore not an optimisation to add later. Without it, the bbox column and small row groups cost size and buy nothing.
Why the bbox column costs 44%
A geometry column stores WKB; the covering bbox adds four doubles per row — 32 bytes before compression.
For small geometries like building footprints, averaging about 7 vertices, that is a large fraction of the row. Measured: 5.06 MB to 7.30 MB.
For large geometries — administrative boundaries with thousands of vertices — the same 32 bytes is negligible, and the feature is nearly free. The overhead is worst exactly where the row count is highest.
Why partitioning and row groups are different tools
Partitioning prunes at the file level using the directory name, so a reader skips files without opening them. It suits low-cardinality keys with even sizes: a region, a year, a tile.
Row groups prune within a file using statistics, so the file is opened and its metadata read. It suits high-cardinality or continuous values, and it is finer-grained.
Use both: partition by the coarse key you always filter on, and set row groups within each partition for everything else.
Edge cases or notes
- Check
num_row_groups. One group means no pushdown. - Sort spatially before writing, or the statistics are useless.
write_covering_bbox=Trueis required forbbox=on read, and costs 44% here.geopandas.to_parquethas nopartition_cols. Write partitions with agroupby.- Even partition sizes matter more than the partitioning scheme.
- Very small partitions cost more than they save — aim for tens of megabytes.
- Column selection is free pushdown. Reading three columns of thirty reads a tenth of the file.
- Make fallbacks loud, or "pushdown" quietly becomes "read everything".
Internal links
- GeoParquet and columnar storage explained — why the format is fast
- Cloud-native geospatial explained — the pattern this belongs to
- How to scale a GeoPandas job with dask-geopandas — reading partitions in parallel
- How to read spatial data from S3 and other object storage — serving partitions remotely
- GIS vector file formats compared — the size comparison in context
- How to choose chunk and tile sizes that actually help — the raster equivalent
- How to reduce GIS file size in Python without wrecking the data — other size levers
- How to process a very large GeoPackage in chunks with Python — when the format is not columnar
FAQ
Why does bbox= fail on my GeoParquet file?
The file has no covering-bbox column. Rewrite it with write_covering_bbox=True; it costs about 44% more space for small geometries.
Why is my filter not any faster?
Almost certainly one row group. Check pq.ParquetFile(path).metadata.num_row_groups and set row_group_size when writing.
Do I need to sort my data before writing?
Yes, for spatial pushdown to work. Without a spatial sort every row group's bounding box covers the whole dataset and none can be skipped.
How small should row groups be?
Tens of thousands of rows is a good starting point. Small enough that statistics discriminate, large enough that per-group overhead stays low.
How do I write partitioned GeoParquet?
geopandas.to_parquet has no partition_cols. Group by the key and write one file per partition into a key=value directory, or use dask_geopandas.
Is GeoParquet really that much smaller?
Measured on 59,391 building polygons: 5.06 MB against 16.59 MB for GeoPackage, 27.04 MB for GeoJSON and 50.31 MB for a shapefile.
Should I partition or use row groups?
Both. Partition on the coarse key you always filter on; use row groups within each partition for everything else.