Batch processing
Running one operation across a whole folder without babysitting it.
Everything on Spatial Workflow about batch processing, in the order that usually works: understand the idea, then run the task, then fix it when it goes wrong. 34 guides in total.
5 learning paths run through this topic — follow one if you would rather work through a route than pick a single guide.
🧭 Start with the concepts
The ideas the how-to guides below assume you already have.
- The Anatomy of a Batch Job: Discover, Apply, Report batch job structure python
- Failure Policy in Batch Processing: Fail Fast, Skip or Quarantine batch error handling policy
- Choosing the Unit of Work in a Batch Job: File, Layer, Feature or Tile batch unit of work gis
- Incremental Processing: How a Batch Job Knows What Changed incremental batch processing gis
⚙️ Then do the task
Step-by-step walkthroughs with code you can copy and adapt.
- How to Batch Process Multiple Shapefiles in Python batch process shapefiles python
- How to Batch Process a Folder of GIS Files in Python: The Complete Workflow batch process gis files python
- How to Batch Convert Shapefiles to GeoPackage in Python batch convert shapefile to geopackage python
- How to Batch Clip Many Layers to a Single Boundary in Python batch clip shapefiles boundary python geopandas
- How to Merge Many Shapefiles into One File in Python merge multiple shapefiles python geopandas
- How to Batch Export a GeoDataFrame to Multiple Formats in Python export geodataframe multiple formats python
- How to Speed Up Batch GIS Jobs with Parallel Processing in Python parallel processing gis python batch
- How to Build a Resumable Batch GIS Job in Python resumable batch job python gis
- How to Log and Summarise Errors in a Batch GIS Job in Python python logging batch gis errors report
- How to Batch Process Rasters with Rasterio in Python batch process rasters rasterio python
- How to Process GIS Files in Nested Subfolders and Mirror the Output Tree process files subfolders python recursive mirror
- How to Batch Process Layers in QGIS with PyQGIS batch process layers pyqgis
- How to Add a Progress Bar and ETA to a Long GIS Batch Job python progress bar batch job
- How to Build an Inventory of a GIS Data Folder in Python gis data folder inventory python
- How to Add a Dry-Run Mode to a Batch GIS Script dry run mode python script
- How to Package GIS Deliverables into Zipped Bundles with Python package gis deliverables zip python
- How to Process a Very Large GeoPackage in Chunks with Python process large geopackage in chunks
- How to Process Only the Files That Changed Since the Last Run process only changed files python
- How to Split a Large Layer into Tiles for Batch Processing split layer into tiles python
- How to Enrich Many Files Against One Reference Layer batch spatial join reference layer
- How to Merge and Mosaic Rasters in Python with Rasterio merge rasters rasterio mosaic
- How to Load a Folder of Shapefiles into PostGIS with Python load shapefiles into postgis python
- How to Batch-Generate a Map Image for Every Region in Python batch generate maps python
🔧 When it breaks
The errors this topic produces, with the cause explained rather than just the command.
- Batch Script Stops at the First Bad File: How to Keep It Running batch script stops on error
- Python glob Is Not Finding All My Shapefiles: How to Fix It glob not finding files python
- Batch Output Files Keep Overwriting Each Other: How to Fix It batch output files overwriting
- Batch GIS Job Slows Down and Runs Out of Memory: How to Fix It batch job memory leak python
- Parallel GIS Batch Job Hangs or Crashes with multiprocessing: How to Fix It multiprocessing hangs python gis
- Batch Job Gives Different Results Each Run: How to Fix It gis batch non deterministic output
- OSError: Too Many Open Files in a Batch GIS Job too many open files python gis
Learning paths through this topic
A curated route rather than a list — the guides in the order that builds on itself.
-
Raster Analysis
Read, reproject, clip and summarise raster data in Python.
Open path → -
Spatial SQL & PostGIS
Move the heavy lifting from Python into the database.
Open path → -
Making Maps
Turn a GeoDataFrame into a map someone else can read.
Open path → -
Batch Processing
Process many files at once, reliably.
Open path → -
QGIS Automation
Automate QGIS with PyQGIS and Processing models.
Open path →
Related topics
Subjects that keep coming up alongside this one, most overlap first.