Batch processing
Running one operation across a whole folder without babysitting it.
A batch job is a loop with opinions. Find the files, do one thing to each, write the output somewhere traceable — and survive the one file in two hundred that is corrupt, empty, in a different CRS or has a column the others do not.
That survival is the whole subject. A loop with no try/except stops at file seven, four hours in, and the run is worth nothing. A loop that records the failure and carries on finishes the other 199 and leaves you a one-file retry. From there the questions are about discovery, mirrored output paths, deciding on the unit of work, and making the run resumable so a crash costs minutes rather than the afternoon.
The pages below cover the pattern, the variations on it — nested folders, mixed formats, several outputs per input — and the specific ways a folder of real data breaks it.
🧭 Start with the concepts
The ideas the how-to guides below assume you already have.
⚙️ Then do the task
Step-by-step walkthroughs with code you can copy and adapt.
- How to Batch Process Multiple Shapefiles in Python
- How to Batch Process a Folder of GIS Files in Python: The Complete Workflow
- How to Batch Convert Shapefiles to GeoPackage in Python
- How to Batch Clip Many Layers to a Single Boundary in Python
- How to Merge Many Shapefiles into One File in Python
- How to Batch Export a GeoDataFrame to Multiple Formats in Python
Show every how-to guide in this topic
- How to Speed Up Batch GIS Jobs with Parallel Processing in Python
- How to Build a Resumable Batch GIS Job in Python
- How to Log and Summarise Errors in a Batch GIS Job in Python
- How to Batch Process Rasters with Rasterio in Python
- How to Process GIS Files in Nested Subfolders and Mirror the Output Tree
- How to Batch Process Layers in QGIS with PyQGIS
- How to Add a Progress Bar and ETA to a Long GIS Batch Job
- How to Build an Inventory of a GIS Data Folder in Python
- How to Add a Dry-Run Mode to a Batch GIS Script
- How to Package GIS Deliverables into Zipped Bundles with Python
- How to Process a Very Large GeoPackage in Chunks with Python
- How to Process Only the Files That Changed Since the Last Run
- How to Split a Large Layer into Tiles for Batch Processing
- How to Enrich Many Files Against One Reference Layer
- How to Merge and Mosaic Rasters in Python with Rasterio
- How to Load a Folder of Shapefiles into PostGIS with Python
- How to Batch-Generate a Map Image for Every Region in Python
- How to Clip and Tile a Point Cloud in Python
- How to Export a Static Tile Pyramid for Offline Use
- How to Batch Geocode Thousands of Addresses in Python
- How to Make a Map Series with Consistent Symbology
- How to Open Hundreds of NetCDF Files as One Dataset
- How to Split a Catchment into Sub-basins at Many Outlets
🔧 When it breaks
The errors this topic produces, with the cause explained rather than just the command.
- Batch Script Stops at the First Bad File: How to Keep It Running
- Python glob Is Not Finding All My Shapefiles: How to Fix It
- Batch Output Files Keep Overwriting Each Other: How to Fix It
- Batch GIS Job Slows Down and Runs Out of Memory: How to Fix It
- Parallel GIS Batch Job Hangs or Crashes with multiprocessing: How to Fix It
- Batch Job Gives Different Results Each Run: How to Fix It
Show every fix in this topic
Learning paths through this topic
A curated route rather than a list — the guides in the order that builds on itself.
Related topics
Subjects that keep coming up alongside this one, most overlap first.