Performance & scale
Spatial indexes, parallelism, memory, and datasets too big to load at once.
Everything on Spatial Workflow about performance & scale, in the order that usually works: understand the idea, then run the task, then fix it when it goes wrong. 67 guides in total.
17 learning paths run through this topic — follow one if you would rather work through a route than pick a single guide.
🧭 Start with the concepts
The ideas the how-to guides below assume you already have.
- Spatial Indexes Explained: R-trees and Why Spatial Joins Are Fast spatial index r-tree explained
- Threads, Processes and the GIL: Parallelism in Python GIS Explained python gil threads processes gis
- Spatial SQL or GeoPandas? Choosing Where the Work Happens spatial sql vs geopandas
- PostGIS Spatial Indexes Explained: GiST and the Two-Phase Filter postgis gist spatial index
- Why GeoPandas Is Slow: The Four Real Bottlenecks why is geopandas slow
- GeoParquet and Columnar Storage Explained: Why It Reads So Much Faster geoparquet explained python
- Nearest-Neighbour Joins Explained: Distance, Ties and Search Radius nearest neighbour join explained
- Lazy Loading Explained: Why xarray Reads Nothing Until You Ask lazy loading xarray explained
- Chunked Arrays and Zarr Explained zarr chunked arrays explained
- Columnar or Row Storage: Why DuckDB Is Fast on Wide Tables columnar row spatial storage
- How DuckDB Uses (and Does Not Use) a Spatial Index duckdb spatial index
- DuckDB's Memory Model: Streaming, Spilling and Limits duckdb memory model
- HTTP Caching for Spatial Data: ETags, Max-Age and Invalidation http caching spatial data
- Reruns and State Explained: Why Your Map App Redraws Everything streamlit rerun state map
- What a Map App Can Afford to Send to the Browser map app performance budget
- Geohash Explained: Prefixes, Precision and the Edge Problem geohash explained precision
- S2 Cells Explained: Squares on a Cube and When They Beat Hexagons s2 geometry cells explained
⚙️ Then do the task
Step-by-step walkthroughs with code you can copy and adapt.
- How to Speed Up Batch GIS Jobs with Parallel Processing in Python parallel processing gis python batch
- How to Cache Pipeline Steps So Only Changed Data Is Reprocessed cache gis pipeline steps python
- How to Reduce GIS File Size in Python Without Wrecking the Data reduce gis file size python
- How to Process a Very Large GeoPackage in Chunks with Python process large geopackage in chunks
- How to Split a Large Layer into Tiles for Batch Processing split layer into tiles python
- How to Read a Large PostGIS Table into Python Without Running Out of Memory read large postgis table python
- How to Build a Small, Fast Python GIS Docker Image small python gis docker image
- How to Replace a Row Loop with Vectorised GeoPandas Operations vectorise geopandas loop
- How to Use the Spatial Index Directly in GeoPandas (sindex) geopandas sindex spatial index
- How to Profile a Slow Python GIS Script and Find the Real Bottleneck profile slow python gis script
- How to Build a Distance Matrix Between Two Layers in Python distance matrix geopandas
- How to Measure Network Distance for Many Origin-Destination Pairs network distance matrix python
- How to Choose Chunk and Tile Sizes That Actually Help choose chunk and tile sizes
- How to Scale a GeoPandas Job with dask-geopandas dask geopandas scale python
- How to Build an Offline Geocoder from Open Address Data offline geocoder python
- How to Aggregate Millions of Points into a Grid with DuckDB duckdb aggregate millions of points
- How to Benchmark DuckDB Against GeoPandas Honestly benchmark duckdb geopandas
- How to Add Bounding Box and Attribute Filters to a Spatial API spatial api bbox filter
- How to Paginate a Large Feature API in Python paginate feature api python
- How to Set Cache Headers on a Spatial API and Tile Service cache headers spatial api
- How to Stream Large Query Results from a Spatial API stream large api response
- How to Cache Spatial Data in a Map App cache spatial data streamlit
- How to Render a Million Points in a Map App with pydeck pydeck million points
- How to Open Hundreds of NetCDF Files as One Dataset xarray open_mfdataset
- How to Join Two Point Datasets on an H3 Index Instead of a Spatial Join join on h3 index
- How to Use H3 in DuckDB for Grid Aggregation at Scale duckdb h3 extension
- How to Index Data by Quadkey and Web Map Tile quadkey python tile index
- How to Compact an H3 Cell Set to Store a Region Cheaply h3 compact cells
- How to Choose Facility Locations with a p-Median Model in Python p-median python pulp
- How to compute distance to the coast for many points distance to coast python
🔧 When it breaks
The errors this topic produces, with the cause explained rather than just the command.
- How to Speed Up GeoPandas: Tips for Large Datasets geopandas performance optimization
- Fixing Memory Errors in GeoPandas When Working with Large Files geopandas memory error
- Batch GIS Job Slows Down and Runs Out of Memory: How to Fix It batch job memory leak python
- Parallel GIS Batch Job Hangs or Crashes with multiprocessing: How to Fix It multiprocessing hangs python gis
- OSError: Too Many Open Files in a Batch GIS Job too many open files python gis
- My PostGIS Spatial Query Is Slow: How to Fix It postgis spatial query slow
- GIS Tests Run Out of Memory or Time Out in CI: How to Fix It ci out of memory gis tests
- GeoPandas .apply() Takes Hours: How to Fix It geopandas apply slow
- Reading a COG from a URL Is Slow or Downloads Everything cog read is slow
- dask-geopandas Uses More Memory or Time Than Plain GeoPandas dask geopandas slower than geopandas
- My GeoJSON Is Too Big for the Browser geojson too big for browser
- Fixing a Map Export That Is Hundreds of Megabytes map export file too large
- Fixing a DuckDB Spatial Join That Never Finishes duckdb spatial join slow
- Fixing DuckDB Out of Memory on a Large Spatial Query duckdb out of memory
- Fixing a Spatial API That Is Slow Under Load spatial api slow under load
- Fixing an API Worker That Runs Out of Memory api worker out of memory
- Fixing a Map App That Takes Ten Seconds to Load map app slow to load
- Fixing an App Whose Memory Grows With Every User streamlit app memory grows
- Fixing open_mfdataset That Is Slow, Hangs or Will Not Combine open_mfdataset slow
- Fixing a Location-Allocation Model That Never Solves pulp p-median slow
Learning paths through this topic
A curated route rather than a list — the guides in the order that builds on itself.
-
Spatial Statistics
Measure the pattern instead of eyeballing the map.
Open path → -
Networks & Routing
Distance along the street, not across the rooftops.
Open path → -
Spatial SQL & PostGIS
Move the heavy lifting from Python into the database.
Open path → -
GIS Data Cleaning
Turn messy spatial data into analysis-ready datasets.
Open path → -
Batch Processing
Process many files at once, reliably.
Open path → -
Automation & Pipelines
Build repeatable, config-driven GIS pipelines.
Open path → -
Cloud-Native GIS
Read the part you need, not the file it lives in.
Open path → -
Publishing Web Maps
Get the data into a browser without shipping the whole dataset.
Open path → -
Geocoding & Addresses
Address text is data too — and it is the messiest data in the file.
Open path → -
Cartography & Map Design
The map is correct. Now make it readable.
Open path → -
DuckDB for Spatial Data
A spatial database with no server, reading the file you already have.
Open path → -
Serving Spatial Data
The dataset is finished. Now other people need it over HTTP.
Open path → -
Maps, Apps & Dashboards
Let the reader ask their own question — within a budget.
Open path → -
Gridded & Climate Data
A raster with a time axis, a calendar and forty years in one file.
Open path → -
H3 & Global Grids
Give every point a cell id and half the geometry work becomes a join.
Open path → -
Location & Accessibility
Who can reach what — and where the next site should go.
Open path → -
Coastal & Marine
The shoreline is a choice of datum, not a line on a map.
Open path →
Related topics
Subjects that keep coming up alongside this one, most overlap first.