How to Containerise a Python GIS Pipeline with Docker

On this page

Problem statement

The pipeline works on your laptop. On the server it does not — and the failures are specific. Built for the companion page on image size, a slim image installed rasterio without complaint and then failed on import:

ImportError: libexpat.so.1: cannot open shared object file: No such file or directory

Python GIS packages are thin wrappers over C libraries — GDAL, GEOS, PROJ — and those libraries live outside the virtualenv. requirements.txt pins geopandas==1.1.4; it says nothing about which GDAL is installed, whether PROJ can find its database, or which drivers were compiled in. Two machines with identical pip freeze output can behave differently: the pyogrio wheel used on this page has no PostgreSQL driver, while the GDAL in the base image it runs on does.

A container fixes the layer pip cannot reach. It packages the native libraries, the Python packages, the environment variables and your code into one image that runs identically on your laptop, the server, and CI.

Where the friction usually is:

a Dockerfile copied from a tutorial that no longer builds — on the current GDAL base, useradd -u 1000 fails because the uid is taken
rebuilding everything on every code change because the layer order is wrong
data baked into the image instead of mounted at run time
files written as root inside the container that nobody can delete outside it
logs that stay empty while the job runs

Quick answer

Start from a maintained GDAL base image, install Python deps in their own layer, copy code last:

base on ghcr.io/osgeo/gdal:ubuntu-small-* — GDAL, PROJ and GEOS already built, pinned by digest
install requirements into a virtualenv before copying code, so the dependency layer caches
run as a non-root user with your host UID
mount data at run time; never COPY datasets into the image
pass configuration and secrets as environment variables or mounted files

FROM ghcr.io/osgeo/gdal:ubuntu-small-3.13.3

ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1 PIP_NO_CACHE_DIR=1
RUN apt-get update && apt-get install -y --no-install-recommends python3-pip python3-venv \
 && rm -rf /var/lib/apt/lists/*
RUN python3 -m venv /venv
ENV PATH="/venv/bin:$PATH"

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY src/ ./src/

USER 1000
ENTRYPOINT ["python", "-m", "src.pipeline"]

docker build -t gis-pipeline:latest .
docker run --rm -u "$(id -u):$(id -g)" \
  -v "$PWD/data:/data" -v "$PWD/configs:/configs:ro" \
  gis-pipeline:latest --config /configs/daily.yml

Copying requirements.txt before src/ is the single most valuable line: a code change then reuses the cached dependency layer. Measured for this page's image, a rebuild after a code change took 0.5 s against 24 s from nothing (Example 1).

What the image actually contains

Four layers, and only the top one changes when you edit code.

What you get — from The data used on this page

1,333 NUTS 3 regions in 36 countries; share aged 65+ in 2023: 2.7–36.0%
{'intermediate': 619, 'rural': 438, 'urban': 276} 1,332 distinct names

Base image, pinned dependencies, layers, user, mounts

Six decisions — the base alone decides between 407 MB and 2.11 GB before any Python is installed.

Choose a base image that already has GDAL

Building GDAL from source is slow, fragile and almost never necessary.

Base	Size, measured	Notes
`ghcr.io/osgeo/gdal:ubuntu-small-3.13.3`	407 MB	official, common drivers including PostgreSQL
`ghcr.io/osgeo/gdal:ubuntu-full-3.13.3`	2.11 GB	every driver the project builds
`python:3.12-slim` + wheels	538 MB with rasterio	wheels bundle their own GDAL; add `libexpat1`

Pin the tag, and for scheduled work the digest. :latest means your build is not reproducible, and a GDAL minor release can change driver behaviour.

FROM ghcr.io/osgeo/gdal:ubuntu-small-3.13.3@sha256:64250faf833c06d4b21afce4c27190039ba7ab58d70f0eebc87cf77d929c0b40

Check what you actually got — Example 1 prints the version and the drivers that matter to this pipeline:

docker run --rm ghcr.io/osgeo/gdal:ubuntu-small-3.13.3 gdalinfo --version
docker run --rm ghcr.io/osgeo/gdal:ubuntu-small-3.13.3 ogrinfo --formats

Measured in Example 1 for GDAL 3.13.3.

Pin the Python dependencies properly

# requirements.txt — exact versions
geopandas==1.1.4
rasterio==1.5.1
pyyaml==6.0.3

For a tamper-evident build, generate hashes with pip-compile --generate-hashes and install with --require-hashes.

Know which GDAL each package uses. Installed from wheels on a GDAL base, rasterio and pyogrio load their own bundled GDAL (3.12.4 here) and ignore the base's (3.13.3). That is fine as long as nothing needs a driver only the base has — and this pipeline does: its PostGIS load goes through the base image's ogr2ogr, because the wheel has no PostgreSQL driver. Either keep that split deliberate, as here, or compile the bindings against the base with --no-binary, as the image-size page does.

Order the layers so the cache works

Docker caches per instruction and invalidates everything after the first change.

# ── rarely changes ──────────────────────────────────────────────
FROM ghcr.io/osgeo/gdal:ubuntu-small-3.13.3
RUN apt-get update && apt-get install -y --no-install-recommends \
      python3-pip python3-venv \
 && rm -rf /var/lib/apt/lists/*
RUN python3 -m venv /venv

# ── changes when dependencies change ────────────────────────────
WORKDIR /app
COPY requirements.txt .
RUN /venv/bin/pip install -r requirements.txt

# ── changes on every commit ─────────────────────────────────────
COPY src/ ./src/

Add a .dockerignore, or the build context includes your data, your secrets and your .git folder:

.git/
.venv/
**/__pycache__/
data/
secrets/
logs/
.env

Run as a real user

Files written by a root container are owned by root on the host, which is a daily annoyance and a security smell.

The usual recipe — useradd -m -u 1000 gis — no longer builds on the current GDAL base. Its Ubuntu 26.04 already has a user ubuntu with uid 1000, and measured, useradd stopped with UID 1000 is not unique. Create a user only when the uid is free:

ARG UID=1000
RUN id -u ${UID} >/dev/null 2>&1 || useradd --create-home --uid ${UID} gis
USER ${UID}

docker build --build-arg UID=$(id -u) -t gis-pipeline .
docker run --rm -u "$(id -u):$(id -g)" -v "$PWD/data:/data" gis-pipeline --config /configs/daily.yml

Mount data, do not bake it in

docker run --rm \
  -v "$PWD/data:/data:rw" \
  -v "$PWD/configs:/configs:ro" \
  -e GIS_DATA_ROOT=/data \
  gis-pipeline:latest --config /configs/daily.yml

Data in the image makes it enormous, stale the moment it is built, and impossible to share. Mount read-only where you can — :ro on the config volume prevents an accidental write to something version-controlled.

For cloud storage, GDAL's virtual filesystems avoid the mount entirely:

docker run --rm \
  -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_REGION \
  -e GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR \
  gis-pipeline:latest --input /vsis3/my-bucket/parcels.gpkg

Set the environment GDAL and PROJ expect

ENV PYTHONUNBUFFERED=1 \
    GDAL_NUM_THREADS=ALL_CPUS \
    CPL_VSIL_CURL_CACHE_SIZE=200000000 \
    VSI_CACHE=TRUE \
    OGR_GEOMETRY_ACCEPT_UNCLOSED_RING=NO

PYTHONUNBUFFERED=1 is not optional. Python block-buffers stdout when it is not a terminal: measured, a container printing a line every half second showed nothing in docker logs after four seconds, and nine lines with the variable set (Example 1).

GDAL_CACHEMAX does not need setting to protect the container: GDAL reads the container's memory limit and sizes its default cache from it — 51 MB under a 1 GB limit, measured on the image-size page. Set it when a job needs more.

Verify the image before you trust it

# a build-time smoke test — fails the build, not the 2 a.m. run
RUN python -c "import geopandas, rasterio; from pyproj import CRS; \
    assert CRS.from_epsg(27700).to_epsg() == 27700; \
    print('smoke ok: geopandas', geopandas.__version__, '| GDAL', rasterio.__gdal_version__)"

A PROJ database that cannot resolve EPSG:27700 is the classic broken-image symptom, and this catches it at build time rather than in production. Example 4 goes further and tests the built image from outside.

A container for a NUTS 3 regions pipeline

The data used on this page

The "districts" are Europe's NUTS 3 regions — 1,333 of them once joined to data — from Eurostat GISCO at 1:1 million, with GISCO's urban–rural typology as the category, and the share of the population aged 65 and over for 2019 to 2023 from the Eurostat table demo_r_pjanind3 as rate_2019 … rate_2023. Eurostat refuses the default python-requests user agent, hence the header.

import math
import os

import geopandas as gpd
import pandas as pd
import requests

YEARS = [2019, 2020, 2021, 2022, 2023]


def eurostat(table, **filters):
    """A Eurostat table as a Series indexed by every dimension."""
    stat = requests.get(f"https://ec.europa.eu/eurostat/api/dissemination/statistics/1.0/data/{table}",
                        params={"format": "JSON", **filters},
                        headers={"User-Agent": "spatialworkflow.io guide"}, timeout=120).json()
    codes = [sorted(stat["dimension"][d]["category"]["index"], key=stat["dimension"][d]["category"]["index"].get)
             for d in stat["id"]]
    values = [stat["value"].get(str(i)) for i in range(math.prod(stat["size"]))]
    return pd.Series(values, index=pd.MultiIndex.from_product(codes, names=stat["id"]), dtype=float)


if not os.path.exists("districts.gpkg"):
    nuts = gpd.read_file("https://gisco-services.ec.europa.eu/distribution/v2/nuts/geojson/"
                         "NUTS_RG_01M_2021_4326_LEVL_3.geojson")
    older = (eurostat("demo_r_pjanind3", indic_de="PC_Y65_MAX", unit="PC",
                      sinceTimePeriod=YEARS[0], untilTimePeriod=YEARS[-1])
             .droplevel(["freq", "indic_de", "unit"]).unstack("time"))
    older.columns = [f"rate_{year}" for year in older.columns]
    kind = {1: "urban", 2: "intermediate", 3: "rural"}
    (nuts.assign(category=nuts["URBN_TYPE"].map(kind))
         .rename(columns={"NUTS_ID": "code", "CNTR_CODE": "country", "NAME_LATN": "name"})
         [["code", "country", "name", "category", "geometry"]]
         .merge(older, left_on="code", right_index=True)
         .dropna(subset=[f"rate_{YEARS[-1]}"])
         .to_file("districts.gpkg"))

districts = gpd.read_file("districts.gpkg")
print(f"{len(districts):,} NUTS 3 regions in {districts.country.nunique()} countries; "
      f"share aged 65+ in {YEARS[-1]}: {districts[f'rate_{YEARS[-1]}'].min():.1f}–"
      f"{districts[f'rate_{YEARS[-1]}'].max():.1f}%")
print(districts.category.value_counts().to_dict(),
      f"{districts.name.nunique():,} distinct names")

for folder in ("src", "tests", "data", "configs", "secrets"):
    os.makedirs(folder, exist_ok=True)
districts[districts["country"] == "PT"].to_file("data/portugal.gpkg")

1,333 NUTS 3 regions in 36 countries; share aged 65+ in 2023: 2.7–36.0%
{'intermediate': 619, 'rural': 438, 'urban': 276} 1,332 distinct names

The pipeline clips Portugal's regions to the mainland, writes a GeoPackage, and loads it into PostGIS when a database is configured:

%%writefile src/pipeline.py
"""Clip Portugal's NUTS 3 regions to the mainland; write a GeoPackage, and load PostGIS if configured."""
import argparse
import os
import subprocess

import geopandas as gpd
import yaml
from shapely.geometry import box


def main(argv=None):
    parser = argparse.ArgumentParser()
    parser.add_argument("--config", required=True)
    args = parser.parse_args(argv)
    with open(args.config) as handle:
        config = yaml.safe_load(handle)

    root = os.environ.get("GIS_DATA_ROOT", "/data")
    regions = gpd.read_file(os.path.join(root, config["input"]))
    mainland = regions.clip(box(*config["bbox"])).to_crs(config["crs"])
    output = os.path.join(root, config["output"])
    os.makedirs(os.path.dirname(output), exist_ok=True)
    mainland.to_file(output)
    print(f"{len(regions)} regions in, {len(mainland)} written to {output}")

    if os.environ.get("PGHOST"):
        # the wheels' GDAL has no PostgreSQL driver; the base image's ogr2ogr does
        with open("/run/secrets/db_password") as secret:
            password = secret.read().strip()
        subprocess.run(["ogr2ogr", "-f", "PostgreSQL", "PG:", output, "-nln", "mainland_regions",
                        "-overwrite", "-lco", "GEOMETRY_NAME=geom"],
                       env={**os.environ, "PGPASSWORD": password}, check=True)
        print("loaded into PostGIS as mainland_regions")


if __name__ == "__main__":
    main()

%%writefile src/__init__.py
# makes src/ a package, so `python -m src.pipeline` works

%%writefile configs/daily.yml
input: portugal.gpkg
output: out/mainland.gpkg
bbox: [-9.6, 36.9, -6.1, 42.2]
crs: 3763

%%writefile requirements.txt
geopandas==1.1.4
rasterio==1.5.1
pyyaml==6.0.3

%%writefile .dockerignore
.git/
.venv/
**/__pycache__/
data/
secrets/
logs/
.env

Example 1: a production Dockerfile

%%writefile Dockerfile
# syntax=docker/dockerfile:1
FROM ghcr.io/osgeo/gdal:ubuntu-small-3.13.3@sha256:64250faf833c06d4b21afce4c27190039ba7ab58d70f0eebc87cf77d929c0b40

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_NO_CACHE_DIR=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=1 \
    DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y --no-install-recommends \
      python3-pip python3-venv \
 && rm -rf /var/lib/apt/lists/*

# a venv rather than --break-system-packages: the system Python stays the distribution's
RUN python3 -m venv /venv
ENV PATH="/venv/bin:$PATH"

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY src/ ./src/

# a build-time smoke test — fails the build, not the 2 a.m. run
RUN python -c "import geopandas, rasterio; from pyproj import CRS; \
    assert CRS.from_epsg(27700).to_epsg() == 27700; \
    print('smoke ok: geopandas', geopandas.__version__, '| GDAL', rasterio.__gdal_version__)"

# Ubuntu 26.04 images already have uid 1000 (user "ubuntu"): add a user only if the uid is free
ARG UID=1000
RUN id -u ${UID} >/dev/null 2>&1 || useradd --create-home --uid ${UID} gis
USER ${UID}

ENTRYPOINT ["python", "-m", "src.pipeline"]
CMD ["--help"]

Built, rebuilt after a code change, and run as the host user:

import subprocess
import time


def sh(*args, check=True):
    return subprocess.run(args, capture_output=True, text=True, check=check)


started = time.perf_counter()
build = sh("docker", "build", "--no-cache", "--progress=plain", "-t", "gis-pipeline:test", ".")
cold = time.perf_counter() - started
print(next(line.split(" ", 2)[2] for line in build.stderr.splitlines() if "smoke ok" in line and "print" not in line))
with open("src/pipeline.py", "a") as out:
    out.write("# a code change\n")
started = time.perf_counter()
sh("docker", "build", "-q", "-t", "gis-pipeline:test", ".")
print(f"built from nothing in {cold:.0f} s, after a code change in {time.perf_counter() - started:.1f} s")

for image in ("ghcr.io/osgeo/gdal:ubuntu-small-3.13.3", "ghcr.io/osgeo/gdal:ubuntu-full-3.13.3", "gis-pipeline:test"):
    sh("docker", "pull", "-q", image, check=False)
    print(f"{image:40} {sh('docker', 'images', image, '--format', '{{.Size}}').stdout.strip()}")

drivers = sh("docker", "run", "--rm", "--entrypoint", "sh", "gis-pipeline:test", "-c",
             "gdalinfo --version; ogrinfo --formats | grep -c . ; ogrinfo --formats | grep -i postgresql;"
             "python -c \"import pyogrio; d = pyogrio.list_drivers(); "
             "print('the pyogrio wheel:', pyogrio.__gdal_version_string__, len(d), 'drivers, PostgreSQL:', 'PostgreSQL' in d)\"")
print(drivers.stdout.strip())

run = sh("docker", "run", "--rm", "-u", f"{os.getuid()}:{os.getgid()}", "-v", f"{os.getcwd()}/data:/data",
         "-v", f"{os.getcwd()}/configs:/configs:ro", "gis-pipeline:test", "--config", "/configs/daily.yml")
print(run.stdout.strip())

smoke ok: geopandas 1.1.4 | GDAL 3.12.4
built from nothing in 24 s, after a code change in 0.5 s
ghcr.io/osgeo/gdal:ubuntu-small-3.13.3   407MB
ghcr.io/osgeo/gdal:ubuntu-full-3.13.3    2.11GB
gis-pipeline:test                        899MB
GDAL 3.13.3 "Iowa City", released 2026/08/13
74
  PostgreSQL -vector- (rw+u): PostgreSQL/PostGIS
  PGDUMP -vector- (w+v): PostgreSQL SQL dump (*.sql)
the pyogrio wheel: 3.12.4 64 drivers, PostgreSQL: False
25 regions in, 23 written to /data/out/mainland.gpkg

The page's earlier Dockerfile installed build-essential and then purged it with apt-get purge in a later instruction. That removes nothing from the image — the earlier layer still holds the files — as two builds that differ only in the purge show. And the other two claims above, checked on the base image — useradd with uid 1000, and what docker logs shows while a job is running:

purge = {"purged": "&& apt-get purge -y build-essential python3-dev && apt-get autoremove -y", "kept": ""}
for name, tail in purge.items():
    with open("Dockerfile.purge", "w") as out:
        out.write("FROM ghcr.io/osgeo/gdal:ubuntu-small-3.13.3\n"
                  "RUN apt-get update && apt-get install -y --no-install-recommends python3-pip python3-dev "
                  "build-essential && rm -rf /var/lib/apt/lists/*\n"
                  f"RUN pip3 install --break-system-packages geopandas==1.1.4 rasterio==1.5.1 {tail}\n")
    sh("docker", "build", "-q", "-f", "Dockerfile.purge", "-t", f"purge-test:{name}", ".")
    print(f"build tools {name:7} in a later RUN: {sh('docker', 'images', f'purge-test:{name}', '--format', '{{.Size}}').stdout.strip()}")
    sh("docker", "image", "rm", f"purge-test:{name}")

added = sh("docker", "run", "--rm", "--entrypoint", "sh", "ghcr.io/osgeo/gdal:ubuntu-small-3.13.3", "-c",
           "getent passwd 1000; useradd -m -u 1000 gis", check=False)
print(added.stdout.strip(), "|", added.stderr.strip())

for env in ([], ["-e", "PYTHONUNBUFFERED=1"]):
    job = sh("docker", "run", "-d", *env, "--entrypoint", "python3", "ghcr.io/osgeo/gdal:ubuntu-small-3.13.3",
             "-c", "import time\nfor i in range(20):\n    print(f'step {i}'); time.sleep(0.5)").stdout.strip()
    time.sleep(4)
    shown = len(sh("docker", "logs", job).stdout.splitlines())
    sh("docker", "wait", job)
    print(f"{' '.join(env) or 'default buffering':22} lines in docker logs after 4 s: {shown}, "
          f"after exit: {len(sh('docker', 'logs', job).stdout.splitlines())}")
    sh("docker", "rm", job)

build tools purged  in a later RUN: 1.29GB
build tools kept    in a later RUN: 1.29GB
ubuntu:x:1000:1000:Ubuntu:/home/ubuntu:/bin/bash | useradd: UID 1000 is not unique
default buffering      lines in docker logs after 4 s: 0, after exit: 20
-e PYTHONUNBUFFERED=1  lines in docker logs after 4 s: 9, after exit: 20

Measured in Examples 1, 2 and 4: each looked right in the Dockerfile and was wrong in the container.

Example 2: docker compose for a pipeline plus PostGIS

%%writefile compose.yaml
services:
  db:
    image: postgis/postgis:17-3.5
    environment:
      POSTGRES_DB: gis
      POSTGRES_USER: gis
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets: [db_password]
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U gis -d gis"]
      interval: 5s
      retries: 10

  pipeline:
    build:
      context: .
      args:
        UID: "${HOST_UID:-1000}"
    image: gis-pipeline:test
    depends_on:
      db: { condition: service_healthy }
    environment:
      GIS_DATA_ROOT: /data
      PGHOST: db
      PGUSER: gis
      PGDATABASE: gis
    secrets: [db_password]
    volumes:
      - ./data:/data
      - ./configs:/configs:ro
    command: ["--config", "/configs/daily.yml"]

secrets:
  db_password:
    file: ./secrets/db_password.txt

volumes:
  pgdata:

The password is a file, mounted as a Docker secret, never an environment variable in the compose file:

import secrets

with open("secrets/db_password.txt", "w") as out:
    out.write(secrets.token_urlsafe(18))
os.chmod("secrets/db_password.txt", 0o600)

print("bash says:", subprocess.run(["bash", "-c", "UID=12345 true"], capture_output=True, text=True).stderr.strip())
env = {**os.environ, "HOST_UID": str(os.getuid())}
started = time.perf_counter()
run = subprocess.run(["docker", "compose", "-p", "swcontain", "run", "--rm", "pipeline"],
                     capture_output=True, text=True, env=env, check=True)
print("\n".join(l for l in run.stdout.splitlines() if "regions" in l or "PostGIS" in l),
      f"({time.perf_counter() - started:.0f} s, database start included)")
count = sh("docker", "compose", "-p", "swcontain", "exec", "-T", "db", "psql", "-U", "gis", "-d", "gis", "-tAc",
           "select count(*) || ' regions, ' || round(sum(st_area(geom)) / 1e6) || ' km²' from mainland_regions")
print("in PostGIS:", count.stdout.strip())
sh("docker", "compose", "-p", "swcontain", "down", "-v")

bash says: bash: line 1: UID: readonly variable
25 regions in, 23 written to /data/out/mainland.gpkg
loaded into PostGIS as mainland_regions (7 s, database start included)
in PostGIS: 23 regions, 88793 km²

The UID reaches the build as HOST_UID, not UID. The obvious UID=$(id -u) docker compose run --rm pipeline fails in bash, where UID is a read-only shell variable: bash prints UID: readonly variable, compose falls back to ${UID:-1000}, and on any machine whose user is not 1000 the files come out with the wrong owner.

A healthcheck plus depends_on: condition: service_healthy removes the classic "database not ready yet" race on the first run — here the pipeline waited for a freshly created database and loaded on the first attempt.

Example 3: run the container from cron or systemd

%%writefile run_container.sh
#!/usr/bin/env bash
# run_container.sh — the nightly job
set -uo pipefail

ROOT="${GIS_ROOT:-/srv/gis}"
IMAGE="${IMAGE:-gis-pipeline:test}"
LOG="$ROOT/logs/pipeline-$(date +%Y%m%d).log"
mkdir -p "$ROOT/logs"

{
  echo "=== start $(date -Is) image=$IMAGE ==="
  docker run --rm \
    -u "$(id -u):$(id -g)" \
    -v "$ROOT/data:/data" \
    -v "$ROOT/configs:/configs:ro" \
    --env-file "$ROOT/job.env" \
    --memory 8g --cpus 4 \
    "$IMAGE" --config /configs/daily.yml
  status=$?
  echo "=== done $(date -Is) exit=$status ==="
} >> "$LOG" 2>&1
exit "$status"

30 2 * * * /srv/gis/run_container.sh

The status is captured on the line after docker run. An earlier version of this script ran under set -e and logged exit=$? from inside its echo — so a failed run killed the script before the "done" line, and a successful one always logged exit=0. Run twice, once with a config that points at a missing file:

import glob
import shutil

for name, source in (("good", "portugal.gpkg"), ("bad", "missing.gpkg")):
    root = f"cron-{name}"
    os.makedirs(f"{root}/configs", exist_ok=True)
    shutil.copytree("data", f"{root}/data", dirs_exist_ok=True)
    with open(f"{root}/configs/daily.yml", "w") as out:
        out.write(open("configs/daily.yml").read().replace("portugal.gpkg", source))
    with open(f"{root}/job.env", "w") as out:
        out.write("GIS_DATA_ROOT=/data\n")
    done = subprocess.run(["bash", "run_container.sh"], env={**os.environ, "GIS_ROOT": os.path.abspath(root)})
    log = open(glob.glob(f"{root}/logs/*.log")[0]).read().splitlines()
    logged = next(word for word in log[-1].split() if word.startswith("exit="))
    print(f"{name}: script exit {done.returncode}; last log line records {logged}")
    if name == "bad":
        print("       ", next(l for l in log if "Error" in l))

good: script exit 0; last log line records exit=0
bad: script exit 1; last log line records exit=1
        pyogrio.errors.DataSourceError: /data/missing.gpkg: No such file or directory

Pinning an immutable tag means the nightly job runs the image you tested, not whatever :latest points at today. --memory turns a runaway job into a contained failure rather than an unresponsive host.

Example 4: a test that the image is sound

%%writefile tests/test_container.py
"""Run against a built image in CI."""
import json
import subprocess

IMAGE = "gis-pipeline:test"


def run(entrypoint, *args):
    # the image's ENTRYPOINT is the pipeline, so anything else needs --entrypoint
    return subprocess.run(["docker", "run", "--rm", "--entrypoint", entrypoint, IMAGE, *args],
                          capture_output=True, text=True, check=True).stdout


def test_gdal_and_proj_are_present():
    versions = json.loads(run("python", "-c",
                              "import rasterio, pyproj, json;"
                              "print(json.dumps({'gdal': rasterio.__gdal_version__,"
                              " 'proj': pyproj.proj_version_str}))"))
    assert versions["gdal"].startswith("3.")


def test_can_reproject():
    out = run("python", "-c",
              "import geopandas as gpd; from shapely.geometry import Point;"
              "g = gpd.GeoDataFrame(geometry=[Point(-3.19, 55.95)], crs=4326).to_crs(27700);"
              "print(round(g.geometry.iloc[0].x))")
    assert 300_000 < int(out.strip()) < 400_000


def test_base_gdal_can_write_postgis():
    assert "PostgreSQL" in run("ogrinfo", "--formats")


def test_runs_as_non_root():
    assert run("id", "-u").strip() != "0"

import sys

!{sys.executable} -m pytest -q -p no:cacheprovider tests/test_container.py 2>&1 | tail -1
print(sh("docker", "image", "rm", "gis-pipeline:test").stdout.splitlines()[0])

4 passed in 1.80s
Untagged: gis-pipeline:test

The page's earlier version of these tests called docker run IMAGE python3 -c …. Against an image whose ENTRYPOINT is the pipeline, those arguments go to the pipeline's argparse, which answers error: the following arguments are required: --config — every test failed. --entrypoint is the fix, and the reason a smoke test belongs in the build as well.

Two environments stacked

A Python GIS environment is really two environments stacked. The upper one is Python packages, which pip and a lock file describe completely. The lower one is native libraries — GDAL with its driver set, GEOS, PROJ with its coordinate-operation database — and nothing in requirements.txt describes it at all. Most "works on my machine" problems in GIS live in that lower layer.

A venv pins Python packages; a container pins everything below them too.

A container captures both. The image is a filesystem with the native libraries, the Python packages, the environment variables and your code already in place, so "it worked in CI" and "it worked on the server" become the same statement. That is the entire argument, and it is a strong one for pipelines that must run unattended.

The build mechanics reward a little care. Docker caches each instruction and invalidates every layer after the first change, so copying requirements.txt and installing before copying src/ is what keeps a code-only rebuild to seconds. The same layering is why cleaning up after the fact does not work: a file deleted in a later RUN is still in the earlier layer, and the image is no smaller.

Two operational details cause most of the day-to-day pain. The first is user identity: a container running as root writes root-owned files into your mounted volume, which you then cannot delete — and a fixed uid only helps if it matches the host's and is free in the image. The second is buffering — Python block-buffers stdout when it is not a terminal, so docker logs stays empty unless PYTHONUNBUFFERED=1 is set.

Finally, keep data out of images and pin your tags. An image is code and environment; data belongs on a mounted volume or in object storage, where it can change without a rebuild. And a pinned, immutable tag is what makes a scheduled run reproducible — :latest quietly turns every night into a new experiment.

Two GDALs in one image

rasterio/pyogrio wheels bundle their own GDAL: on a GDAL base image you have two. Use the base's for what the wheels lack — PostgreSQL here — or compile with --no-binary.
Use a venv rather than --break-system-packages on Ubuntu 23.10 and later, which mark the system Python as externally managed.
Ubuntu 26.04 images already have uid 1000. Create a user only if the uid is free.
UID is read-only in bash. Pass the host uid to compose under another name.
Alpine is a trap for GIS: packages without musllinux wheels, rasterio among them, compile from source.
PROJ_LIB / PROJ_DATA moved: PROJ 9 renamed the variable. Prefer a base image that sets it correctly over setting it yourself.
Multi-arch builds: On Apple Silicon, docker build --platform linux/amd64 matters when the server is x86; emulated GDAL is slow.
Layer size: every RUN creates a layer, so clean up in the same instruction that created the files — purging later saves nothing.
Time zone and locale: Containers default to UTC and the C locale, which can change date parsing and string sorting compared with your laptop.

Internal links

FAQ

Which base image should I use for a Python GIS pipeline?

ghcr.io/osgeo/gdal:ubuntu-small-<version> when the pipeline needs the base's GDAL tools or drivers — 407 MB measured, with the PostgreSQL driver the wheels lack. python:3.12-slim plus wheels (and libexpat1) when GeoPandas and rasterio are all you need.

Why is my image several gigabytes?

Usually a full GDAL base you do not need, build tools left in an earlier layer, or data copied in. Use the -small base, install and clean up in the same RUN, and add a .dockerignore.

Why does every build reinstall all the dependencies?

Because code is copied before requirements.txt. Copy the requirements file and install first, then copy the source, so the dependency layer stays cached.

How do I stop the container writing root-owned files?

Run with -u "$(id -u):$(id -g)", or build with your uid as an argument — and create the user only if that uid is not already taken, as it is on Ubuntu 26.04 images.

Should data go inside the image?

No. Mount it as a volume or read it from object storage with GDAL's /vsis3/, /vsigs/ handlers. Baking data in bloats the image and makes it stale immediately.

Why do I see no logs until the run finishes?

Python buffers stdout when it is not a terminal. Measured, docker logs showed nothing four seconds into a job printing twice a second. Set ENV PYTHONUNBUFFERED=1 in the Dockerfile, or run with python3 -u.

How do I keep the container's GDAL and Python packages consistent?

Pin both: a base image by digest and exact Python versions, hash-pinned where you can. Know which GDAL each package loads — the wheels bring their own — and add a build-time smoke test that reprojects a point.