How to Run a Python GIS Pipeline in CI with GitHub Actions
On this page
Problem statement
The pipeline is tested, containerised and scheduled — on one machine, by one person. Which means:
- a change is only validated when someone remembers to run the tests locally
- "works on my machine" is still the only evidence that it works
- the nightly job runs from a laptop that is sometimes closed
- nobody notices that
geopandas1.1 changed a default until the output is wrong - the deliverable is built by hand, so no two builds are quite the same
Continuous integration moves all of that onto a machine that starts clean every time. For GIS work it does something more specific too: it proves the pipeline runs in an environment that is not yours, which is the only real test of whether the dependency pinning is honest.
The friction is real, though. Geospatial dependencies are heavy, CI runners have no GDAL by default, spatial tests may need a database, and jobs that download data are slow and flaky.
Quick answer
Run the tests on every push, and the pipeline on a schedule, in the same container the server uses:
- cache the Python environment so installs do not dominate the run
- install GDAL via the container image or
apt, not by compiling - run the fast test suite on pull requests, the full one on
main - run the pipeline on a
schedule:trigger and upload the output as an artifact - keep credentials in repository secrets, never in the workflow file
# .github/workflows/ci.yml
name: CI
on:
push:
branches: [main]
pull_request:
schedule:
- cron: "30 2 * * *" # 02:30 UTC — note: UTC, not your local time
jobs:
test:
runs-on: ubuntu-latest
container: ghcr.io/osgeo/gdal:ubuntu-small-3.9.2
steps:
- uses: actions/checkout@v4
- name: Install Python dependencies
run: |
apt-get update && apt-get install -y --no-install-recommends python3-pip python3-venv
python3 -m venv /opt/venv # the image's Debian numpy blocks a plain pip install
/opt/venv/bin/pip install -r requirements.txt -r requirements-dev.txt
echo /opt/venv/bin >> "$GITHUB_PATH"
- name: Run tests
run: pytest -q -m "not db"
- name: Verify the geospatial stack
run: |
python3 -c "
import geopandas as gpd
from shapely.geometry import Point
g = gpd.GeoDataFrame(geometry=[Point(-3.19, 55.95)], crs=4326).to_crs(27700)
assert 300_000 < g.geometry.iloc[0].x < 400_000
print('geopandas', gpd.__version__, 'ok')"Running the job inside the GDAL image gives the job the GDAL command-line tools (ogr2ogr, gdalwarp) at a pinned version. Two things about it were found by running these exact steps locally in ghcr.io/osgeo/gdal:ubuntu-small-3.9.2. First, the obvious install — pip3 install --break-system-packages -r requirements.txt — fails: ERROR: Cannot uninstall numpy 1.26.4, RECORD file not found. Hint: The package was installed by debian. The image ships Debian's NumPy 1.26 for its own Python bindings, and GeoPandas 1.2 needs NumPy 2. A virtual environment, as above, avoids the conflict (19 s to install). Second, the Python stack does not use the image's GDAL at all: in that 3.9.2 image, pyogrio reported GDAL 3.12.4 and PROJ 9.8.1 — the libraries bundled in its wheels. The image's GDAL serves the command-line tools; the wheels serve GeoPandas.
What each trigger is for
What you get — from Example 5: check a workflow before you push it
3 complete workflows; actionlint exit 0 (no findings)
container steps exit 0 | geopandas 1.2.0 | pyogrio GDAL 3.12.4 | GDAL 3.9.2, released 2024/08/13
GDAL on the runner, caching, services and schedules
Get GDAL onto the runner
Three approaches, in order of preference:
# 1. run the job inside an image that already has it ← best
jobs:
test:
runs-on: ubuntu-latest
container: ghcr.io/osgeo/gdal:ubuntu-small-3.9.2# 2. rely on wheels — geopandas, pyogrio, rasterio and shapely all ship binaries
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/setup-python@v5
with: { python-version: "3.12", cache: pip }
- run: pip install -r requirements.txt # no system GDAL needed# 3. install the distro packages, when you need the CLI tools too
- run: |
sudo apt-get update
sudo apt-get install -y gdal-bin libgdal-dev
echo "GDAL_VERSION=$(gdal-config --version)" >> "$GITHUB_ENV"Option 2 is enough for a pure GeoPandas pipeline and is the fastest to set up; option 1 is right when the pipeline also shells out to ogr2ogr or gdalwarp, or when you want CI and production to share one image. Do not try to make the image's Python bindings and the pip stack share an environment: forcing NumPy 2 in with --ignore-installed installs, but then the image's from osgeo import gdal_array fails with numpy.core.multiarray failed to import, because the bindings were compiled for NumPy 1.
Cache the dependencies
Installing the geospatial stack takes a couple of minutes; caching makes it seconds.
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
cache-dependency-path: requirements*.txt
# cache test fixtures or reference data that rarely change
- name: Cache reference data
uses: actions/cache@v4
with:
path: data/ref
key: refdata-${{ hashFiles('scripts/fetch_reference.py') }}-v1
- name: Fetch reference data
run: python scripts/fetch_reference.py # a no-op on a cache hitInclude a version suffix (-v1) in cache keys you may need to bust manually — changing it is easier than deleting caches through the UI.
Split the fast job from the slow one
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12", cache: pip }
- run: pip install ruff
- run: ruff check src tests
- run: ruff format --check src tests
test:
needs: lint
runs-on: ubuntu-latest
container: ghcr.io/osgeo/gdal:ubuntu-small-3.9.2
steps:
- uses: actions/checkout@v4
- run: |
apt-get update && apt-get install -y --no-install-recommends python3-pip python3-venv
python3 -m venv /opt/venv # the image's Debian numpy blocks a plain pip install
/opt/venv/bin/pip install -r requirements.txt -r requirements-dev.txt
echo /opt/venv/bin >> "$GITHUB_PATH"
- run: pytest -q -m "not db" --junitxml=report.xml
- uses: actions/upload-artifact@v4
if: always()
with: { name: test-report, path: report.xml }Linting first fails a badly formatted pull request in fifteen seconds instead of after a four-minute test run.
Add PostGIS as a service when you need it
test-db:
runs-on: ubuntu-latest
services:
postgis:
image: postgis/postgis:16-3.4
env:
POSTGRES_DB: gis
POSTGRES_USER: gis
POSTGRES_PASSWORD: gis
ports: ["5432:5432"]
options: >-
--health-cmd "pg_isready -U gis"
--health-interval 5s --health-timeout 5s --health-retries 10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12", cache: pip }
- run: pip install -r requirements.txt -r requirements-dev.txt
- name: Run database tests
env:
PGHOST: localhost
PGUSER: gis
PGPASSWORD: gis
PGDATABASE: gis
run: pytest -q -m dbThe health check is what removes the "connection refused on the first run" flake — the steps do not start until Postgres is actually accepting connections.
Test across versions where it matters
matrix:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python: ["3.11", "3.12", "3.13"]
geopandas: ["1.0.1", "1.1.0"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "${{ matrix.python }}", cache: pip }
- run: |
pip install -r requirements-dev.txt
pip install "geopandas==${{ matrix.geopandas }}"
- run: pytest -q -m "not db"fail-fast: false matters here: you want to see which combinations broke, not just that one did.
Run the pipeline on a schedule and keep the output
nightly:
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
container: ghcr.io/osgeo/gdal:ubuntu-small-3.9.2
timeout-minutes: 60
steps:
- uses: actions/checkout@v4
- run: |
apt-get update && apt-get install -y --no-install-recommends python3-pip python3-venv
python3 -m venv /opt/venv # the image's Debian numpy blocks a plain pip install
/opt/venv/bin/pip install -r requirements.txt
echo /opt/venv/bin >> "$GITHUB_PATH"
- name: Run the pipeline
env:
PGHOST: ${{ secrets.PGHOST }}
PGUSER: ${{ secrets.PGUSER }}
PGPASSWORD: ${{ secrets.PGPASSWORD }}
PGDATABASE: ${{ secrets.PGDATABASE }}
GIS_DATA_ROOT: ${{ github.workspace }}/data
run: python3 -m src.pipeline --config configs/daily.yml
- name: Upload outputs
uses: actions/upload-artifact@v4
with:
name: pipeline-output-${{ github.run_number }}
path: |
data/out/**
logs/runs/*.json
retention-days: 30
- name: Summarise the run
if: always()
run: |
python3 - <<'PY' >> "$GITHUB_STEP_SUMMARY"
import json, glob
latest = sorted(glob.glob("logs/runs/*.json"))[-1]
rec = json.load(open(latest))
print(f"### Run `{rec['run_id']}` — {rec['status']}\n")
print(f"- duration: {rec['duration_s']}s")
for o in rec["outputs"]:
print(f"- output: `{o['path']}` — {o.get('features', '?')} features")
PY$GITHUB_STEP_SUMMARY renders Markdown on the run's page, so the feature counts are visible without downloading anything.
Add a scheduled-workflow keepalive and alerting
- name: Notify on failure
if: failure()
uses: actions/github-script@v7
with:
script: |
github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title: `Nightly pipeline failed (run ${context.runNumber})`,
body: `The scheduled pipeline failed.\n\n${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`,
labels: ["pipeline", "failure"],
})Two scheduling facts worth knowing, both from GitHub's documentation: scheduled workflows run in UTC by default, and in a public repository they are disabled automatically after 60 days without repository activity — so a quiet repo silently stops running its nightly job.
A complete workflow and its variations
Example 1: a complete workflow file
name: GIS pipeline
on:
push: { branches: [main] }
pull_request:
schedule: [{ cron: "30 2 * * *" }]
workflow_dispatch:
inputs:
config:
description: Config file to run
default: configs/daily.yml
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
env:
PYTHON_VERSION: "3.12"
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12", cache: pip }
- run: pip install ruff
- run: ruff check src tests && ruff format --check src tests
test:
needs: lint
runs-on: ubuntu-latest
services:
postgis:
image: postgis/postgis:16-3.4
env: { POSTGRES_DB: gis, POSTGRES_USER: gis, POSTGRES_PASSWORD: gis }
ports: ["5432:5432"]
options: >-
--health-cmd "pg_isready -U gis" --health-interval 5s --health-retries 10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12", cache: pip }
- run: pip install -r requirements.txt -r requirements-dev.txt
- name: Smoke-test the geospatial stack
run: |
python -c "
import geopandas as gpd, pyproj, shapely
from shapely.geometry import Point
g = gpd.GeoDataFrame(geometry=[Point(-3.19, 55.95)], crs=4326).to_crs(27700)
assert 300_000 < g.geometry.iloc[0].x < 400_000
print('geopandas', gpd.__version__, '| proj', pyproj.proj_version_str)"
- name: Tests
env: { PGHOST: localhost, PGUSER: gis, PGPASSWORD: gis, PGDATABASE: gis }
run: pytest -q --junitxml=report.xml --cov=src --cov-report=term-missing
- uses: actions/upload-artifact@v4
if: always()
with: { name: test-report, path: report.xml }
pipeline:
needs: test
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
container: ghcr.io/osgeo/gdal:ubuntu-small-3.9.2
timeout-minutes: 90
steps:
- uses: actions/checkout@v4
- run: |
apt-get update && apt-get install -y --no-install-recommends python3-pip python3-venv
python3 -m venv /opt/venv # the image's Debian numpy blocks a plain pip install
/opt/venv/bin/pip install -r requirements.txt
echo /opt/venv/bin >> "$GITHUB_PATH"
- name: Run
env:
PGHOST: ${{ secrets.PGHOST }}
PGUSER: ${{ secrets.PGUSER }}
PGPASSWORD: ${{ secrets.PGPASSWORD }}
PGDATABASE: ${{ secrets.PGDATABASE }}
run: python3 -m src.pipeline --config "${{ inputs.config || 'configs/daily.yml' }}"
- uses: actions/upload-artifact@v4
with:
name: output-${{ github.run_number }}
path: |
data/out/**
logs/runs/*.json
retention-days: 30concurrency with cancel-in-progress stops three pushes in five minutes from queueing three full runs.
Example 2: build and publish the container image
image:
needs: test
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions: { contents: read, packages: write }
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/metadata-action@v5
id: meta
with:
images: ghcr.io/${{ github.repository }}
tags: |
type=raw,value=latest
type=sha,format=long
type=raw,value={{date 'YYYYMMDD'}}
- uses: docker/build-push-action@v6
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
cache-from: type=gha
cache-to: type=gha,mode=maxA date-tagged, immutable image per merge is what the scheduled server job should pin to.
Example 3: fail the build when the output changes unexpectedly
- name: Check output against the golden summary
run: |
python3 - <<'PY'
import json, sys
import geopandas as gpd
gdf = gpd.read_file("data/out/parcels.gpkg")
actual = {
"features": len(gdf),
"crs": gdf.crs.to_string(),
"bounds": [round(float(v), 1) for v in gdf.total_bounds],
}
expected = json.load(open("tests/golden/parcels_summary.json"))
if actual != expected:
print("::error::pipeline output changed")
print(f"expected: {expected}")
print(f"actual : {actual}")
sys.exit(1)
print("output matches golden summary")
PY::error:: renders as a proper annotation on the run, so the reason is visible without opening the log.
Example 4: a manual re-run with parameters
on:
workflow_dispatch:
inputs:
region:
description: Region to process
type: choice
options: [all, north, south, east, west]
default: all
dry_run:
description: Plan only, write nothing
type: boolean
default: true
jobs:
run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: |
python -m src.pipeline \
--region "${{ inputs.region }}" \
${{ inputs.dry_run && '--dry-run' || '' }}Combining workflow_dispatch with a dry-run default gives non-developers a safe button to press.
Example 5: check a workflow before you push it
A workflow's first real run is usually its first test. Two checks catch most mistakes locally: actionlint, which knows the workflow schema, the expression syntax and the runner labels, and runs shellcheck on every run: script; and running the container steps in the same image. This is how the workflows on this page were checked — the script reads them out of this page as published; in your repository, point it at .github/workflows:
import html
import re
import subprocess
from pathlib import Path
import requests
page = requests.get("https://www.spatialworkflow.io/gis-pipeline-github-actions-ci/", timeout=60).content.decode("utf-8")
blocks = [html.unescape(re.sub(r"<[^>]+>", "", b)) # drop the syntax highlighting
for b in re.findall(r'<code class="language-yaml">(.*?)</code>', page, re.S)]
workflows = [b for b in blocks if b.lstrip().startswith(("# .github", "name:", "on:"))]
folder = Path("ci/.github/workflows")
folder.mkdir(parents=True, exist_ok=True)
for i, text in enumerate(workflows):
(folder / f"workflow_{i}.yml").write_text(text)
subprocess.run(["git", "init", "-q"], cwd="ci") # actionlint looks for a repository
lint = subprocess.run(["uv", "tool", "run", "--from", "actionlint-py", "actionlint", "-no-color"],
cwd="ci", capture_output=True, text=True)
print(len(workflows), "complete workflows; actionlint exit", lint.returncode, lint.stdout.strip() or "(no findings)")
Path("ci/requirements.txt").write_text("geopandas\npyogrio\npytest\n")
steps = """set -eo pipefail
apt-get update -qq >/dev/null && apt-get install -y -qq --no-install-recommends python3-pip python3-venv >/dev/null
python3 -m venv /opt/venv && /opt/venv/bin/pip install -q -r requirements.txt
/opt/venv/bin/python -c "import geopandas as gpd, pyogrio; from shapely.geometry import Point
g = gpd.GeoDataFrame(geometry=[Point(-3.19, 55.95)], crs=4326).to_crs(27700)
assert 300_000 < g.geometry.iloc[0].x < 400_000
print('geopandas', gpd.__version__, '| pyogrio GDAL', pyogrio.__gdal_version_string__)"
gdalinfo --version"""
run = subprocess.run(["docker", "run", "--rm", "-v", f"{Path('ci').resolve()}:/w", "-w", "/w",
"ghcr.io/osgeo/gdal:ubuntu-small-3.9.2", "bash", "-c", steps],
capture_output=True, text=True)
print("container steps exit", run.returncode, "|", " | ".join(run.stdout.strip().splitlines()[-2:]))3 complete workflows; actionlint exit 0 (no findings)
container steps exit 0 | geopandas 1.2.0 | pyogrio GDAL 3.12.4 | GDAL 3.9.2, released 2024/08/13
Planting an error shows the linter is doing real work: a typo in runs-on: ubuntu-latst is reported as an unknown runner label, and an unknown key under a job as a syntax error. What neither check can do is exercise GitHub itself — secrets, the schedule, permissions — so the first run on GitHub is still worth watching.
A machine that has never seen your laptop
CI is a second machine that has never seen your laptop. That is the whole value: it starts from a clean checkout and a clean environment, so anything that only worked because of something installed years ago fails immediately, in public, with a log.
For geospatial work the environment question is sharper than for ordinary Python, because the dependencies extend below the language. The two robust answers are the same as everywhere else in this stack: rely on wheels, which bundle their own GDAL and are enough for a GeoPandas pipeline, or run the job inside the same GDAL image the pipeline uses in production — remembering that inside it the Python packages still use the GDAL in their wheels, and the image's own GDAL serves the command-line tools. Compiling GDAL in CI is possible and almost never worth the build time it adds to every run.
Structuring the workflow around triggers keeps it fast enough to stay enabled. A pull request wants lint plus the fast tests — under two minutes, or people start merging without waiting. A push to main can afford the database tests and an image build. A schedule runs the actual pipeline. A workflow_dispatch gives you a button for re-runs and one-off parameters. Bundling all of that into one job that runs on every event is the most common reason teams end up ignoring a permanently red CI.
Artifacts are what make CI more than a test runner for a data pipeline. Uploading the output and the run record turns each scheduled execution into something reviewable: the feature counts are in the step summary, the GeoPackage is downloadable for thirty days, and a golden-summary check fails the run when the numbers move unexpectedly. That is a data-quality gate, and it costs a few lines of YAML.
Two operational details bite people specifically with scheduled workflows. Cron in Actions is UTC, so a "02:30" job runs an hour off your local time for half the year. And GitHub disables scheduled workflows in repositories with no activity for 60 days — a quiet pipeline repo stops running without any notification at all, which is worth a calendar reminder or a periodic commit.
Schedules, permissions and ephemeral runners
- Schedules are UTC and imprecise: GitHub documents that scheduled runs can be delayed at busy times — the start of every hour especially — and under enough load some queued jobs are dropped. Schedule at an odd minute, and do not depend on the exact time.
- Scheduled workflows in public repositories are auto-disabled after 60 days of inactivity: Push something occasionally, or re-enable it manually.
GITHUB_TOKENpermissions default to read-only: Grantpackages: writeorcontents: writeexplicitly per job when you need them.- Artifacts are not free: They count against storage. Set
retention-days, and do not upload multi-gigabyte rasters routinely. - Runners are ephemeral and modest: the standard Linux runner has 4 CPUs and 16 GB of RAM for public repositories, 2 CPUs and 8 GB for private ones, and 14 GB of disk either way (GitHub's documented figures). Large geospatial jobs need chunking, a larger runner, or a self-hosted one.
- Container jobs run as root: Files created in the workspace are root-owned, which matters if a later step runs as another user.
pull_requestfrom a fork has no secrets: By design. Split jobs so fork PRs run the tests that do not need credentials.
Internal links
- How to Test a GIS Pipeline with pytest
- How to Containerise a Python GIS Pipeline with Docker
- How to Handle Credentials and Secrets in an Automated GIS Job
- How to Schedule a Python GIS Script to Run Automatically
- How to Make a GIS Workflow Reproducible in Python
- How to Get Alerted When an Automated GIS Job Fails
FAQ
How do I get GDAL onto a GitHub Actions runner?
For a GeoPandas pipeline you usually do not need to: geopandas, pyogrio, rasterio and shapely ship wheels with GDAL, GEOS and PROJ inside. Run the job inside ghcr.io/osgeo/gdal:ubuntu-small-<version>, with the Python packages in a virtual environment, when you also need the command-line tools; or apt-get install gdal-bin on the plain runner.
Should the pipeline itself run in CI?
For small to medium jobs, yes: a scheduled workflow plus uploaded artifacts is a complete, free scheduler with logs and alerting. For long or memory-hungry jobs, use CI for tests and images, and run the pipeline on your own infrastructure from a pinned image.
How do I test against PostGIS?
Add postgis/postgis as a service container with a pg_isready health check, and pass the connection settings as environment variables. The health check is what stops the first-run connection flake.
Why did my scheduled workflow stop running?
Either it is a public repository with no activity for 60 days, in which case GitHub disabled it, or the cron is in UTC and running at a different local time than you expected. Under heavy load GitHub can also delay or drop a scheduled run.
How do I keep CI fast?
Cache pip and any downloaded reference data, split lint from tests, run only the fast tests on pull requests, and use concurrency with cancel-in-progress so superseded runs stop immediately.
Where do credentials go?
Repository or environment secrets, referenced as $ and passed via env: on the step that needs them. Never in the workflow file, and remember that pull requests from forks do not receive secrets.
How do I know what the nightly run produced?
Write a run record, upload it and the outputs as an artifact, and print a summary into $GITHUB_STEP_SUMMARY so the counts appear on the run page without downloading anything.