Pipelines & automation
Config-driven, scheduled, observable workflows that run without you.
Everything on Spatial Workflow about pipelines & automation, in the order that usually works: understand the idea, then run the task, then fix it when it goes wrong. 38 guides in total.
2 learning paths run through this topic — follow one if you would rather work through a route than pick a single guide.
12
Concepts
18
How-To guides
8
Fixes
🧭 Start with the concepts
The ideas the how-to guides below assume you already have.
- How to Make a GIS Workflow Reproducible in Python reproducible gis workflow python
- What Is a GIS Data Pipeline? The Anatomy Explained what is a gis data pipeline
- Idempotency Explained: Why a GIS Job Must Be Safe to Re-run idempotency data pipeline explained
- Configuration vs Code: What Belongs in a Config File configuration vs code
- Logs, Metrics and Alerts: Observability for GIS Pipelines observability data pipeline logs metrics
- Cron, systemd, Airflow or CI? Choosing a Scheduler for GIS Jobs choosing a scheduler for python jobs
- What to Test in a GIS Pipeline (and What You Cannot) what to test gis pipeline
- The Anatomy of a Batch Job: Discover, Apply, Report batch job structure python
- Failure Policy in Batch Processing: Fail Fast, Skip or Quarantine batch error handling policy
- Repair, Reject or Flag? Choosing What Cleaning Should Do gis data cleaning decisions
- Choosing the Unit of Work in a Batch Job: File, Layer, Feature or Tile batch unit of work gis
- Incremental Processing: How a Batch Job Knows What Changed incremental batch processing gis
⚙️ Then do the task
Step-by-step walkthroughs with code you can copy and adapt.
- How to Automate GIS Workflows with Python automate gis workflow python
- How to Build a GIS Data Pipeline in Python: The Complete Workflow gis data pipeline python
- How to Chain GIS Processing Steps into a Reusable Pipeline in Python chain gis processing steps python
- How to Drive a GIS Pipeline from a YAML Config File in Python yaml config gis pipeline python
- How to Turn a GIS Script into a Command-Line Tool with argparse python gis script command line argparse
- How to Schedule a Python GIS Script to Run Automatically schedule python gis script cron
- How to Add Retries and Timeouts to an Automated GIS Job in Python python retry timeout gis job
- How to Validate Pipeline Inputs and Outputs Automatically in Python validate gis pipeline inputs outputs
- How to Cache Pipeline Steps So Only Changed Data Is Reprocessed cache gis pipeline steps python
- How to Get Alerted When an Automated GIS Job Fails alert gis job failure python
- How to Containerise a Python GIS Pipeline with Docker docker python gis pipeline
- How to Test a GIS Pipeline with pytest test gis pipeline pytest
- How to Record Run Metadata and Data Lineage in a GIS Pipeline data lineage python gis pipeline
- How to Handle Credentials and Secrets in an Automated GIS Job python secrets credentials gis job
- How to Run a Python GIS Pipeline in CI with GitHub Actions github actions python gis pipeline
- How to Process Only the Files That Changed Since the Last Run process only changed files python
- How to Update a PostGIS Table from a GeoDataFrame Without Duplicates postgis upsert geodataframe
- How to Profile a Slow Python GIS Script and Find the Real Bottleneck profile slow python gis script
🔧 When it breaks
The errors this topic produces, with the cause explained rather than just the command.
- Python GIS Script Works Manually but Not from Cron: How to Fix It cron python script not running
- ModuleNotFoundError in a Scheduled GIS Job: Fixing the Wrong Python Environment modulenotfounderror scheduled script
- Relative Paths Break When a GIS Script Runs Automatically: How to Fix It relative path fails in cron python
- YAML Config File Will Not Load: Common Errors in a Python GIS Pipeline yaml config error python
- My GIS Pipeline Fails Silently: Fixing Swallowed Errors and Wrong Exit Codes python script fails silently
- GIS Tests Pass Locally but Fail in CI: How to Fix It gis tests fail in ci
- pytest Cannot Import My GIS Modules: How to Fix It pytest modulenotfounderror src
- GeoPandas Cannot Connect to PostGIS: How to Fix It geopandas postgis connection error
Learning paths through this topic
A curated route rather than a list — the guides in the order that builds on itself.
Related topics
Subjects that keep coming up alongside this one, most overlap first.