How to Make a GIS Workflow Reproducible in Python

A reviewer asks a simple question: "can you regenerate the figures from last year's report?" You still have the script. You still have the data — probably; there is a folder called final_v3 and one called final_v3_actual. You run it, and GeoPandas has moved on two minor versions, the make_valid behaviour changed, the supplier's boundary file was updated in place six months ago, and the numbers come out different. Nothing was done wrong, exactly. The workflow was simply never built to be repeated. Reproducibility is the discipline of making sure that "run it again" produces the same answer — this year, next year, and on someone else's machine.

Problem statement

You cannot reliably reproduce your own results. The reasons compound:

The environment moved — a library upgrade changed a result and nothing recorded which version produced the original.
The data moved — an input file was overwritten in place, so the version used back then no longer exists anywhere.
The parameters are gone — the threshold used for that run was a literal in a script that has since been edited.
The code is ambiguous — "the script" exists in three copies with slightly different logic and no version history.
Randomness — sampling, clustering, or a tie-break in a join returns a different answer each run.

The goal: a run that records — and can restore — the exact code, environment, inputs, and parameters that produced a given output, so the same command yields the same numbers a year later.

Quick answer

Pin five things and record all five with every run: the environment (locked dependency versions), the code (a commit hash), the inputs (immutable, checksummed files), the parameters (a config file), and any randomness (a fixed seed). Write them into a manifest stored beside the output.

Five layers of reproducibility: environment, code, data, parameters, and randomness, each pinned and recorded. — Miss any one layer and "run it again" stops being a reliable question.

import hashlib, json, platform, subprocess, sys
from datetime import datetime, timezone
from pathlib import Path

def file_hash(path: Path, chunk=1 << 20) -> str:
    h = hashlib.sha256()
    with Path(path).open("rb") as fh:
        for block in iter(lambda: fh.read(chunk), b""):
            h.update(block)
    return h.hexdigest()

def git_commit() -> str:
    try:
        out = subprocess.run(["git", "rev-parse", "HEAD"],
                             capture_output=True, text=True, check=True).stdout.strip()
        dirty = subprocess.run(["git", "status", "--porcelain"],
                               capture_output=True, text=True).stdout.strip()
        return out + ("-dirty" if dirty else "")
    except Exception:
        return "unknown"

def write_manifest(run_dir: Path, cfg: dict, inputs, outputs, stats: dict):
    import geopandas as gpd, pandas as pd, shapely, pyproj, fiona
    manifest = {
        "run_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "code": {"commit": git_commit(), "entrypoint": sys.argv[0]},
        "environment": {
            "python": sys.version.split()[0],
            "platform": platform.platform(),
            "geopandas": gpd.__version__,
            "pandas": pd.__version__,
            "shapely": shapely.__version__,
            "pyproj": pyproj.__version__,
            "fiona": fiona.__version__,
            "gdal": fiona.__gdal_version__,
        },
        "config": json.loads(json.dumps(cfg, default=str)),
        "inputs": [{"path": str(p), "sha256": file_hash(p),
                    "bytes": Path(p).stat().st_size} for p in inputs],
        "outputs": [{"path": str(p), "sha256": file_hash(p)} for p in outputs],
        "stats": stats,
    }
    (run_dir / "manifest.json").write_text(json.dumps(manifest, indent=2))
    return manifest

Step-by-step solution

Pin the environment, do not just record it

Recording versions tells you what happened; pinning them lets you get it back. A lock file captures the whole resolved dependency tree, not just the packages you named.

# uv — fast, lock-file first
uv lock                       # writes uv.lock with every transitive pin
uv sync                       # recreate the exact environment anywhere

# pip-tools
pip-compile requirements.in   # → requirements.txt with pinned versions and hashes
pip-sync requirements.txt

# conda / mamba — the right choice when GDAL is involved
conda env export --no-builds > environment.yml

For GIS specifically, GDAL is the version that matters most and the one people forget, because it comes in through fiona or pyogrio rather than as a direct dependency. It is the layer that actually reads your files, and its behaviour on edge cases genuinely changes between releases.

import fiona, pyproj
print(fiona.__gdal_version__, pyproj.proj_version_str)   # e.g. 3.9.2  9.4.1

pyproj's PROJ version deserves the same attention: transformation grids improve between releases, so the same to_crs() between two datums can shift coordinates by centimetres — enough to change which side of a boundary a point falls on.

Treat input data as immutable

The most common cause of an irreproducible GIS result is an input file that was updated in place. Adopt one rule: downloaded data is never modified and never overwritten. New version, new folder, dated.

data/
  raw/
    2026-01-15/parcels.gpkg      ← never edited, never replaced
    2026-07-02/parcels.gpkg
  interim/                       ← regenerable, safe to delete
  processed/
    2026-08-01/parcels_clean.gpkg

Then checksum what you read, so the manifest can prove which version a run used:

inputs = [cfg["source"], cfg["boundary"]]
for p in inputs:
    log.info("input %s sha256=%s", p.name, file_hash(p)[:16])

If a run's manifest records a hash that no file on disk now matches, you know immediately that the data changed — which is far better than quietly getting different numbers.

Put the code under version control and record the commit

A commit hash is the most compact possible description of "the code that ran". Recording it costs one subprocess call and makes git checkout <hash> a complete restoration of the logic.

"code": {"commit": git_commit(), "entrypoint": sys.argv[0]}

Flagging a dirty working tree matters as much as the hash. A run whose commit is recorded as a1b2c3d-dirty is honest about being unreproducible, which is the second-best outcome and infinitely better than a hash that misrepresents what actually executed.

Externalise every parameter

A threshold living in a script is a parameter you will overwrite the next time you change it. In a config file it is a versioned, diffable artefact — and a copy of the resolved config goes into the run folder.

def snapshot_config(cfg, run_dir):
    (run_dir / "config_used.json").write_text(
        json.dumps(json.loads(json.dumps(cfg, default=str)), indent=2))

Resolved matters: after merging defaults, environment overlays, and command-line flags, with paths made absolute. That is what ran, and it is often not what any single file on disk says.

Fix every source of randomness

Randomness in GIS work is sneakier than in general data science because much of it is not obviously random.

import random, numpy as np

SEED = 42
random.seed(SEED)
np.random.seed(SEED)

sample = gdf.sample(n=1000, random_state=SEED)              # pandas needs its own
labels = KMeans(n_clusters=5, random_state=SEED).fit_predict(coords)

Beyond explicit sampling, watch for these:

Row order. glob() does not guarantee ordering — always sorted(). A join that produces ties resolves them by order, so unordered input gives different output.
Dictionary and set iteration. Sets have no order; building a layer list from a set can shuffle results between runs.
Parallel workers. Results come back in completion order, not submission order. Sort by a stable key before writing — see parallel batch processing.
Floating-point summation order. Summing areas in a different order can change the last decimal place; irrelevant for a map, visible in a regression test.

Write a manifest with every run

A run manifest recording timestamp, git commit, library versions, input checksums, config, and output counts. — The manifest is the artefact that lets a future colleague — usually you — trust the output.

{
  "run_at": "2026-08-01T02:15:07+00:00",
  "code": { "commit": "9f2c1ab", "entrypoint": "run_pipeline.py" },
  "environment": {
    "python": "3.12.4", "geopandas": "1.0.1", "shapely": "2.0.5",
    "pyproj": "3.6.1", "fiona": "1.9.6", "gdal": "3.9.2"
  },
  "inputs": [
    { "path": "data/raw/2026-07-02/parcels.gpkg", "sha256": "7d3f…", "bytes": 48213504 }
  ],
  "config": { "target_crs": "EPSG:27700", "min_area_m2": 50, "seed": 42 },
  "outputs": [ { "path": "parcels_clean.gpkg", "sha256": "b41a…" } ],
  "stats": { "features_in": 18204, "features_out": 17655, "seconds": 412.7 }
}

Make reruns easy and safe

Reproducibility that requires archaeology does not get used. One command should recreate the environment and rerun a specific configuration.

.PHONY: setup run verify
setup:
	uv sync --frozen

run:
	uv run python run_pipeline.py --config config/prod.yml

verify:
	uv run python tools/verify_manifest.py data/processed/2026-08-01/manifest.json

# tools/verify_manifest.py — do the recorded inputs still exist, unchanged?
import json, sys
from pathlib import Path

manifest = json.loads(Path(sys.argv[1]).read_text())
problems = []
for entry in manifest["inputs"]:
    path = Path(entry["path"])
    if not path.exists():
        problems.append(f"missing input: {path}")
    elif file_hash(path) != entry["sha256"]:
        problems.append(f"input changed since the run: {path}")
if manifest["code"]["commit"].endswith("-dirty"):
    problems.append("code had uncommitted changes when this ran")
print("\n".join(problems) if problems else "manifest verified: inputs unchanged")
sys.exit(1 if problems else 0)

Code examples

Example 1: A reproducibility preamble for any script

Ten lines at the top of the entry point that pin randomness and record the environment.

import os, random, sys
import numpy as np

SEED = int(os.environ.get("SEED", 42))
random.seed(SEED)
np.random.seed(SEED)
os.environ["PYTHONHASHSEED"] = str(SEED)     # affects a fresh interpreter only

def environment_report():
    import geopandas as gpd, shapely, pyproj, fiona, pandas as pd
    return {
        "python": sys.version.split()[0],
        "geopandas": gpd.__version__, "pandas": pd.__version__,
        "shapely": shapely.__version__, "pyproj": pyproj.__version__,
        "proj": pyproj.proj_version_str,
        "fiona": fiona.__version__, "gdal": fiona.__gdal_version__,
    }

print(json.dumps(environment_report(), indent=2))

Example 2: A dated, self-describing run folder

Everything about a run in one directory: the data, the config, the log, the manifest.

from datetime import datetime, timezone
from pathlib import Path

def make_run_dir(base: Path, name: str) -> Path:
    stamp = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H%M%SZ")
    run_dir = base / f"{stamp}_{name}"
    run_dir.mkdir(parents=True, exist_ok=False)     # never reuse a run folder
    (run_dir / "logs").mkdir()
    return run_dir

run_dir = make_run_dir(Path("runs"), "parcels")
# runs/2026-08-01T021507Z_parcels/{manifest.json,config_used.json,logs/,parcels_clean.gpkg}

exist_ok=False is deliberate: a collision means two runs are about to share a folder, and failing is the correct response.

Example 3: Deterministic ordering everywhere it matters

The four ordering fixes that eliminate most "it changed and I don't know why".

files = sorted(src_dir.glob("*.gpkg"))                    # 1. stable file order
gdf = gdf.sort_values("parcel_id").reset_index(drop=True) # 2. stable row order
gdf = gdf[sorted(gdf.columns.drop("geometry")) + ["geometry"]]   # 3. stable columns

results = [f.result() for f in futures]                   # 4. parallel: sort after
results = sorted(results, key=lambda g: g["parcel_id"].iloc[0])

Example 4: Pinning the environment with Docker

When the environment must survive an operating-system upgrade, containerise it.

FROM ghcr.io/osgeo/gdal:ubuntu-small-3.9.2

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --require-hashes -r requirements.txt

COPY . .
ENTRYPOINT ["python", "run_pipeline.py"]

docker build -t parcels-pipeline:2026-08-01 .
docker run --rm -v "$PWD/data:/app/data" parcels-pipeline:2026-08-01 --config config/prod.yml

Tagging the image with a date rather than latest is what makes it an archive rather than a moving target. --require-hashes ensures even a re-published package version cannot change the build.

Example 5: A regression test that catches silent drift

The strongest guarantee is an automated one: a small fixture through the real pipeline, with the numbers asserted.

# tests/test_pipeline_regression.py
import geopandas as gpd
from pipeline import run_pipeline

def test_known_input_gives_known_output(tmp_path):
    cfg = {
        "source": "tests/fixtures/parcels_sample.gpkg",   # committed, immutable
        "output_dir": tmp_path,
        "target_crs": "EPSG:27700",
        "min_area_m2": 50,
        "seed": 42,
    }
    manifest = run_pipeline(cfg)
    assert manifest["features_out"] == 143
    result = gpd.read_file(manifest["output"])
    assert result.crs.to_string() == "EPSG:27700"
    assert round(result["area_m2"].sum(), 1) == 284_913.7

When a library upgrade changes a result, this test fails on the upgrade rather than six months later in a report. Loosen the tolerance rather than deleting the assertion when a change is legitimate — and record why in the commit message.

Explanation

Reproducibility is not one property but five, and a workflow is only as reproducible as its weakest layer. Pinning every library while reading an input that gets overwritten monthly buys nothing. Checksumming the data while the analysis lives in an uncommitted notebook buys nothing either. The five layers — environment, code, data, parameters, randomness — have to be addressed together, which is why the manifest that records all five in one file is the single highest-value habit in this article.

Four causes of irreproducible results — library upgrade, overwritten input, edited script, unseeded randomness — each with its fix. — Every "it gave different numbers this time" traces back to one of four unpinned things.

GIS has a reproducibility hazard that general data analysis does not: the transformation stack underneath your code is itself versioned and improving. PROJ ships better datum-shift grids; GDAL fixes edge cases in drivers; Shapely 2 changed how invalid geometries behave under operations. These are improvements, and they change results. That is exactly why recording the GDAL and PROJ versions matters more than recording your own library's — when a number moves, the first question is which layer moved underneath it, and without a manifest that question takes days instead of minutes.

There is a distinction worth being precise about. Repeatability is getting the same answer on the same machine tomorrow. Reproducibility is a colleague getting the same answer on a different machine. Replicability is a different team reaching the same conclusion with their own data and code. Manifests and seeds give you the first. Lock files, immutable inputs, and committed code give you the second. The third is a scientific question, not an engineering one — but you cannot even attempt it without the first two.

None of this needs to be adopted at once. In practice the order of return is clear: write a manifest (an afternoon, and it makes every later question answerable), then stop overwriting input data (a folder convention), then pin the environment with a lock file, then add a seed and deterministic ordering, and finally add a regression test on a small fixture. Each step is useful alone, and the first one is useful immediately — including for runs that happened before you adopted the rest.

Edge cases or notes

Hashing very large inputs

SHA-256 over a 50 GB raster takes minutes. Hash a header plus a sample of blocks, or record size, modification time, and any internal metadata the format provides. Note in the manifest which method you used, so nobody later mistakes a partial fingerprint for a full one.

Data you do not control

For a WFS or a live database, there is no file to checksum. Record what you can: the query, the timestamp of the fetch, the server's Last-Modified header, and a hash of the fetched result. Then save that fetched snapshot as an immutable input — it is now the reproducible artefact.

Floating-point differences across machines

Identical code and versions can still differ in the last bits across CPU architectures or BLAS builds. Compare with tolerances (pytest.approx, numpy.isclose) rather than exact equality, and reserve byte-for-byte comparison for a single controlled environment such as a container.

Notebooks

Notebooks record output but not execution order, and a notebook that ran cells out of order is not reproducible even by its author. Keep the pipeline in modules under version control and use notebooks to call it, or enforce top-to-bottom execution in CI with jupyter nbconvert --execute.

"-dirty" runs

Sometimes you must run with uncommitted changes. Record it honestly rather than suppressing it, and treat a -dirty manifest as provisional: fine for an exploratory run, not acceptable for a number that goes into a report.

Storage cost of immutable data

Never overwriting inputs means storage grows. That is usually the right trade — disk is cheaper than an irreproducible result — but manage it deliberately: compress older snapshots, move them to cheaper storage, keep the checksums forever even when the bytes are archived offline.

Internal links

FAQ

What does it take to make a GIS workflow reproducible?

Pin and record five things: the environment (a lock file, including the GDAL and PROJ versions), the code (a git commit), the inputs (immutable files with checksums), the parameters (a config file saved with the run), and randomness (a fixed seed plus deterministic ordering). Missing any one of them is enough to break "run it again".

Why do my results change after upgrading GeoPandas or GDAL?

Because the layer underneath your code changed. GDAL fixes driver edge cases, PROJ ships improved datum-transformation grids, and Shapely 2 altered how invalid geometries behave. These are genuine improvements that change output. Recording those versions in a manifest turns a baffling difference into a one-line explanation.

What should a run manifest contain?

The UTC timestamp, the git commit (flagged if the tree was dirty), the Python and library versions including GDAL and PROJ, the checksum and size of every input, the fully resolved config, the checksum of every output, and summary statistics such as feature counts and runtime. Write it next to the output, not in a separate log.

How do I keep input data immutable?

Never write into the folder you downloaded to, and never overwrite a raw file. Save each acquisition under a dated folder, treat those folders as read-only, and checksum the files you read so the manifest records exactly which version a run used. Regenerable intermediates go somewhere clearly disposable.

Is `random.seed()` enough to make a run deterministic?

No. NumPy and pandas have their own generators (np.random.seed, random_state=), and much GIS non-determinism is not "random" at all — unsorted glob() results, set iteration order, and parallel workers returning out of order. Sort file lists, sort rows before writing, and impose a stable order on parallel results.

Do I need Docker for reproducible GIS work?

Not to start. A lock file, immutable inputs, a committed script, and a manifest cover most needs and cost far less. Reach for a container when the environment must survive an OS upgrade, when GDAL is difficult to install consistently across machines, or when someone else must reproduce your work on a system you cannot see.

How do I know reproducibility has broken before it matters?

Add a regression test: run a small committed fixture through the real pipeline in CI and assert on feature counts and a summed measurement. When a dependency upgrade changes a result, the test fails on the upgrade instead of six months later in a report — and you get to decide deliberately whether the new number is the better one.

How to Make a GIS Workflow Reproducible in Python #

Problem statement #

Quick answer #

Step-by-step solution #

Pin the environment, do not just record it #

Treat input data as immutable #

Put the code under version control and record the commit #

Externalise every parameter #

Fix every source of randomness #

Write a manifest with every run #

Make reruns easy and safe #

Code examples #

Example 1: A reproducibility preamble for any script #

Example 2: A dated, self-describing run folder #

Example 3: Deterministic ordering everywhere it matters #

Example 4: Pinning the environment with Docker #

Example 5: A regression test that catches silent drift #

Explanation #

Edge cases or notes #

Hashing very large inputs #

Data you do not control #

Floating-point differences across machines #

Notebooks #

"-dirty" runs #

Storage cost of immutable data #

Internal links #

FAQ #

What does it take to make a GIS workflow reproducible? #

Why do my results change after upgrading GeoPandas or GDAL? #

What should a run manifest contain? #

How do I keep input data immutable? #

Is random.seed() enough to make a run deterministic? #

Do I need Docker for reproducible GIS work? #

How do I know reproducibility has broken before it matters? #