Provenance and lineage explained for spatial workflows

Problem statement

The number in the report is 21,007. Six months later somebody asks where it came from, and the honest answer is: from a notebook that has been edited since, run against a file that has been replaced, using a library that has been upgraded twice. The map is still on the wall, and nobody can reproduce it.

Lineage is the ordered record of what went into a result and what was done to it. Provenance is the broader question it answers: can this output be traced, explained and reproduced? Every other metadata field can be recomputed from the data; lineage cannot, because it describes events that have already happened.

This guide sets out what a lineage record has to contain to be worth keeping, how much detail is enough, and why the checksum you record for a spatial file needs care. The examples record a real run: HM Land Registry's freehold parcels in Rutland counted by ONS ward.

Quick answer

Record, for every output: the inputs with their versions and content hashes, the operations with their parameters, the code version, the environment, and when it ran.

record = {
    "output": "parcels_per_ward.gpkg",
    "inputs": [{"name": ..., "version": ..., "licence": ..., "source": ..., "sha256": ...}],
    "steps": [{"op": ..., "params": {...}}],
    "environment": {"python": ..., "packages": {...}, "libraries": {"GDAL": ..., "PROJ": ..., "GEOS": ...},
                    "git": {"commit": ..., "dirty": ...}, "run_at": ...},
    "output_sha256": ...,
}

Example 2 writes exactly that for a real run and prints it. Everything in that record is either machine-readable at run time or a constant. None of it depends on anyone remembering anything.

Define the output, identify inputs by content, hash content

1. Decide what an "output" is

Lineage attaches to an artefact. If your pipeline writes six files, it writes six lineage records — or one record listing six outputs. What it must not do is describe "the pipeline" in general, because the question people ask is always about a specific file.

2. Identify inputs by content, not by path

data/parcels.gml identifies nothing: the file at that path changes. Record a name, a version or date, and a content hash. The hash is what turns "we used the Rutland extract" into a checkable claim — HM Land Registry republishes the INSPIRE polygons every month under the same URL.

3. Hash the content, not the container

A byte checksum of an output you write yourself is only reproducible if the format is. Write the same 15 wards twice, a couple of seconds apart, to the same file names:

import hashlib, time

def sha256_of(path):
    digest = hashlib.sha256()
    with open(path, "rb") as f:
        for block in iter(lambda: f.read(1 << 20), b""):
            digest.update(block)
    return digest.hexdigest()

writers = {
    "GeoPackage": ("hash_test/wards_g.gpkg", lambda p: wards.to_file(p, driver="GPKG")),
    "GeoJSON": ("hash_test/wards_j.geojson", lambda p: wards.to_file(p, driver="GeoJSON")),
    "FlatGeobuf": ("hash_test/wards_f.fgb", lambda p: wards.to_file(p, driver="FlatGeobuf")),
    "Shapefile .dbf": ("hash_test/wards_s.dbf", lambda p: wards.to_file(p.replace(".dbf", ".shp"))),
    "Parquet": ("hash_test/wards_p.parquet", lambda p: wards.to_parquet(p)),
}
Path("hash_test").mkdir(exist_ok=True)
digests = {}
for attempt in range(2):
    for name, (path, write) in writers.items():
        for old in Path("hash_test").glob(Path(path).stem + ".*"):
            old.unlink()
        write(path)
        digests.setdefault(name, []).append(sha256_of(path))
    time.sleep(2)
for name, (first, second) in digests.items():
    print(f"{name:15} {'identical' if first == second else 'DIFFERENT'}")

header = open("hash_test/wards_s.dbf", "rb").read(4)
print(f".dbf header date: {1900 + header[1]}-{header[2]:02d}-{header[3]:02d}")

GeoPackage      DIFFERENT
GeoJSON         identical
FlatGeobuf      identical
Shapefile .dbf  identical
Parquet         identical
.dbf header date: 2026-09-30

Only the GeoPackage changed: it records its own last-change time. The others are conditionally stable — GeoJSON and FlatGeobuf carry the layer name GDAL takes from the file name, and a shapefile's .dbf header carries the date it was written, so a rewrite tomorrow differs. For inputs you did not create, a byte hash of the file as downloaded is fine and is the right thing; for outputs you create, hash the canonicalised data. How to checksum spatial datasets so you can prove they match has the method.

4. Record parameters that change the answer, not every argument

A parameter belongs in the lineage if a different value would produce a different output that somebody might question: the routing algorithm, the threshold, the resampling method, the CRS, the simplification tolerance, the random seed — and, in the parcel count, the rule that assigns a parcel to one ward. Buffer sizes, yes. Chunk sizes and thread counts, no — they change speed, not results.

5. Pin the code and the environment

A commit hash and a dirty flag, plus the versions of the libraries that do the arithmetic. GDAL, PROJ and GEOS all change results between versions; recording them is what makes "it was different last year" a diagnosable statement.

6. Write the record at the same moment as the data

A lineage record produced by a separate step can be written for a file that was never produced. Write it in the same function that writes the output, from the same variables.

7. Keep it with the artefact and in the log

Beside the file so it travels; in a run log so you can ask questions across runs, such as which outputs used a source you have just discovered was wrong.

8. Make it queryable

The payoff comes when someone reports a problem with an input. A lineage store lets you answer "which published outputs used this file?" in a query instead of a week.

Capturing the environment and recording steps

The data used on this page

HM Land Registry's INSPIRE Index Polygons for Rutland (Open Government Licence, © Crown copyright and database rights), downloaded as the zip the service publishes, and Rutland's 15 wards from the ONS Wards (December 2024) boundaries, saved as the GeoJSON the ONS service returns. Both are hashed exactly as downloaded. The counting code lives in a small git repository, pipeline/, so the record can name a commit.

import io
import json
import subprocess
import zipfile
from pathlib import Path

import geopandas as gpd
import requests

Path("data").mkdir(exist_ok=True)
if not Path("data/Rutland.zip").exists():
    reply = requests.Session().get(
        "https://use-land-property-data.service.gov.uk/datasets/inspire/download/Rutland.zip", timeout=300)
    reply.raise_for_status()
    Path("data/Rutland.zip").write_bytes(reply.content)
if not Path("data/wards.geojson").exists():
    reply = requests.get(
        "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/"
        "Wards_December_2024_Boundaries_UK_BFC/FeatureServer/0/query",
        params={"where": "LAD24CD='E06000017'", "outFields": "WD24CD,WD24NM", "outSR": 27700, "f": "geojson"},
        timeout=120)
    reply.raise_for_status()
    Path("data/wards.geojson").write_bytes(reply.content)
wards = gpd.read_file("data/wards.geojson")

Path("pipeline").mkdir(exist_ok=True)
Path("pipeline/count_parcels.py").write_text(
    '"""Count freehold parcels per ward, assigning each parcel by its representative point."""\n'
    "import geopandas as gpd\n\n"
    "def count(parcels, wards):\n"
    "    points = parcels.set_geometry(parcels.representative_point())\n"
    "    joined = gpd.sjoin(points, wards[['WD24CD', 'WD24NM', 'geometry']], predicate='within')\n"
    "    counts = joined.groupby(['WD24CD', 'WD24NM']).size().rename('parcels').reset_index()\n"
    "    return wards[['WD24CD', 'geometry']].merge(counts, on='WD24CD')\n")
Path("pipeline/.gitignore").write_text("__pycache__/\n")
if not Path("pipeline/.git").exists():
    git = lambda *args: subprocess.run(["git", "-C", "pipeline", *args], check=True, capture_output=True)
    git("init", "-q")
    git("add", "count_parcels.py", ".gitignore")
    git("-c", "user.name=pipeline", "-c", "user.email=pipeline@example.org", "commit", "-q", "-m", "count parcels per ward")
print(len(wards), "wards;", f"{Path('data/Rutland.zip').stat().st_size / 1e6:.1f} MB zip")

15 wards; 5.9 MB zip

Example 1 — capture the environment and the code version

import subprocess, sys, importlib.metadata as md, datetime

def environment(repo="."):
    def version(pkg):
        try:
            return md.version(pkg)
        except md.PackageNotFoundError:
            return None
    try:
        commit = subprocess.check_output(["git", "-C", repo, "rev-parse", "--short", "HEAD"], text=True).strip()
        dirty = bool(subprocess.check_output(["git", "-C", repo, "status", "--porcelain"], text=True).strip())
    except Exception:
        commit, dirty = None, None
    import pyogrio, pyproj, shapely
    return {
        "python": sys.version.split()[0],
        "packages": {p: version(p) for p in ("geopandas", "shapely", "pyproj", "pyogrio", "pandas", "numpy")},
        "libraries": {"GDAL": pyogrio.__gdal_version_string__, "PROJ": pyproj.proj_version_str,
                      "GEOS": shapely.geos_version_string},
        "git": {"repo": repo, "commit": commit, "dirty": dirty},
        "run_at": datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds"),
    }

env = environment("pipeline")
print({k: env[k] for k in ("python", "libraries", "git")})

Path("pipeline/scratch.txt").write_text("an uncommitted edit")
print("after an uncommitted edit, dirty =", environment("pipeline")["git"]["dirty"])
Path("pipeline/scratch.txt").unlink()

{'python': '3.14.4', 'libraries': {'GDAL': '3.12.4', 'PROJ': '9.8.1', 'GEOS': '3.13.1'}, 'git': {'repo': 'pipeline', 'commit': 'b5e194b', 'dirty': False}}
after an uncommitted edit, dirty = True

The dirty flag is what stops a commit hash from vouching for code that was never committed. It is also easy to trip for the wrong reason: without the .gitignore in pipeline/, merely importing count_parcels writes a __pycache__ folder, and every run after the first reports a dirty tree.

Example 2 — a step recorder that cannot be forgotten

import functools, json, pathlib, sys
sys.path.insert(0, "pipeline")
from count_parcels import count

class Lineage:
    def __init__(self, output, repo="."):
        self.record = {"output": str(output), "inputs": [], "steps": [], "environment": environment(repo)}

    def input(self, name, path, version=None, licence=None, source=None):
        self.record["inputs"].append({
            "name": name, "path": str(path), "version": version, "licence": licence, "source": source,
            "sha256": sha256_of(path),
        })

    def step(self, op, **params):
        def wrap(fn):
            @functools.wraps(fn)
            def inner(*a, **kw):
                self.record["steps"].append({"op": op, "params": params})
                return fn(*a, **kw)
            return inner
        return wrap

    def write(self, output_path, store="lineage"):
        self.record["output_sha256"] = sha256_of(output_path)
        pathlib.Path(store).mkdir(exist_ok=True)
        text = json.dumps(self.record, indent=2)
        beside = pathlib.Path(str(output_path) + ".lineage.json")
        beside.write_text(text)                                   # travels with the file
        (pathlib.Path(store) / beside.name).write_text(text)      # and is queryable across runs
        return beside


def run_parcel_count(out="parcels_per_ward.gpkg", rule="representative_point"):
    lin = Lineage(out, repo="pipeline")
    lin.input("hmlr_inspire_rutland", "data/Rutland.zip", version="downloaded " + env["run_at"][:10],
              licence="OGL-UK-3.0", source="use-land-property-data.service.gov.uk")
    lin.input("ons_wards_dec2024_bfc", "data/wards.geojson", version="December 2024",
              licence="OGL-UK-3.0", source="ONS Open Geography Portal")

    @lin.step("read_gml", member="Land_Registry_Cadastral_Parcels.gml")
    def read_parcels():
        with zipfile.ZipFile("data/Rutland.zip") as z:
            z.extract("Land_Registry_Cadastral_Parcels.gml", "data")
        return gpd.read_file("data/Land_Registry_Cadastral_Parcels.gml", columns=["INSPIREID"])

    @lin.step("count_per_ward", assign_by=rule, predicate="within")
    def per_ward(parcels):
        return count(parcels, wards)

    result = per_ward(read_parcels())
    result.to_file(out, layer="parcels_per_ward")
    lin.write(out)
    return result


result = run_parcel_count()
print(result[["WD24NM", "parcels"]].sort_values("parcels", ascending=False).head(3).to_string(index=False))
print(f"{result['parcels'].sum():,} parcels assigned to a ward")

record = json.loads(Path("parcels_per_ward.gpkg.lineage.json").read_text())
print(json.dumps({
    "output": record["output"],
    "inputs": [{k: (v[:12] + "…" if k == "sha256" else v) for k, v in i.items() if k not in ("path", "source")}
               for i in record["inputs"]],
    "steps": record["steps"],
    "code": record["environment"]["git"],
    "libraries": record["environment"]["libraries"],
    "output_sha256": record["output_sha256"][:12] + "…",
}, indent=1, ensure_ascii=False))

                   WD24NM  parcels
                Uppingham     2053
             Oakham South     1981
Braunston & Martinsthorpe     1905
21,007 parcels assigned to a ward
{
 "output": "parcels_per_ward.gpkg",
 "inputs": [
  {
   "name": "hmlr_inspire_rutland",
   "version": "downloaded 2026-09-30",
   "licence": "OGL-UK-3.0",
   "sha256": "b33fece30cea…"
  },
  {
   "name": "ons_wards_dec2024_bfc",
   "version": "December 2024",
   "licence": "OGL-UK-3.0",
   "sha256": "4770091729df…"
  }
 ],
 "steps": [
  {
   "op": "read_gml",
   "params": {
    "member": "Land_Registry_Cadastral_Parcels.gml"
   }
  },
  {
   "op": "count_per_ward",
   "params": {
    "assign_by": "representative_point",
    "predicate": "within"
   }
  }
 ],
 "code": {
  "repo": "pipeline",
  "commit": "b5e194b",
  "dirty": false
 },
 "libraries": {
  "GDAL": "3.12.4",
  "PROJ": "9.8.1",
  "GEOS": "3.13.1"
 },
 "output_sha256": "17459f9fcc2f…"
}

Attaching the record to the function that does the work is what stops the two drifting apart: count_per_ward cannot run without appending its step, with the assignment rule that decides which ward a boundary-straddling parcel lands in.

Example 3 — ask which outputs used a bad input

A second output from the same sources, and a third that uses only the wards, give the store something to search:

def ward_areas():
    lin = Lineage("ward_areas.gpkg", repo="pipeline")
    lin.input("ons_wards_dec2024_bfc", "data/wards.geojson", version="December 2024", licence="OGL-UK-3.0")
    areas = wards.assign(area_km2=wards.area / 1e6)[["WD24CD", "WD24NM", "area_km2", "geometry"]]
    areas.to_file("ward_areas.gpkg")
    lin.write("ward_areas.gpkg")

ward_areas()
run_parcel_count("parcels_per_ward_rerun.gpkg")                   # the same job, run again


def outputs_using(source_name, store="lineage"):
    hits = []
    for f in pathlib.Path(store).glob("*.lineage.json"):
        rec = json.loads(f.read_text())
        for i in rec["inputs"]:
            if i["name"] == source_name:
                hits.append({"output": rec["output"], "input_sha256": i["sha256"][:12],
                             "run_at": rec["environment"]["run_at"]})
    return sorted(hits, key=lambda h: h["output"])

for hit in outputs_using("hmlr_inspire_rutland"):
    print(hit)
print(len(outputs_using("ons_wards_dec2024_bfc")), "outputs used the wards")

{'output': 'parcels_per_ward.gpkg', 'input_sha256': 'b33fece30cea', 'run_at': '2026-09-30T13:37:18+00:00'}
{'output': 'parcels_per_ward_rerun.gpkg', 'input_sha256': 'b33fece30cea', 'run_at': '2026-09-30T13:39:33+00:00'}
3 outputs used the wards

This query is the reason to keep lineage in a directory rather than only beside each file. When HM Land Registry corrects a parcel, the hash tells you which outputs were made before the correction.

Lineage records events that cannot be recomputed

Why lineage cannot be reconstructed

Extent, CRS, schema and feature counts are properties of the data and can be recomputed at any time. Lineage describes events: which file existed at that path on that day, which version of GEOS was installed, what the threshold was before somebody changed it. Once those have passed, nothing in the data records them.

Why versions of libraries belong in the record

Geometry operations are not stable across versions. GEOS has changed the behaviour of buffer, simplify and overlay noding between releases; PROJ has changed default transformation pipelines as new grids shipped; interpolation libraries change their defaults. A result that differs from last year's is usually explained by one of those, and only if they were recorded.

Why the input hash matters more than the input path

The commonest cause of an unreproducible result is that the input changed underneath a stable path. A content hash converts "we used the current extract" into a statement that can be tested against the file you still have, and it is what lets you distinguish "the data changed" from "the code changed".

Why the detail level is a decision

Full provenance — every function call, every intermediate — is expensive to capture and almost never read. The useful level is the one that answers the questions you get: which sources, which versions, which parameters, which code. Record that reliably rather than recording everything unreliably.

Dirty trees, seeds and manual steps

• A dirty working tree invalidates the commit hash. Record the flag.
• Random seeds are parameters. Any stochastic step needs its seed in the record.
• Manual steps must be recorded as manual. "Edited in QGIS by hand" is lineage.
• Downloads need the date and the URL. Remote sources change without notice; the Land Registry URL above serves a different file every month.
• Record failures too. A run that produced nothing still explains a gap in a series.
• Do not put credentials in the record. Record the endpoint, not the token.
• Lineage grows. Rotate or index it; a directory of a million JSON files is not queryable.
• Sign it if it matters. For regulated work, a signed record is worth the extra step.

Internal links

• How to record lineage automatically in a pipeline — the implementation
• How to checksum spatial datasets so you can prove they match — hashing content rather than bytes
• Dataset versioning explained — naming the versions lineage refers to
• You cannot tell which data version produced a map — the failure this prevents
• Spatial metadata explained: what a dataset must tell you — where lineage sits among the other fields
• How to record run metadata and lineage in a GIS pipeline — the pipeline-side pattern
• Reproducible GIS environments explained — pinning what the record names
• How to build a reproducible GIS workflow in Python — the wider practice

FAQ

What is the difference between provenance and lineage?

Lineage is the record of inputs, operations and versions. Provenance is the property that record gives you: the ability to trace, explain and reproduce an output.

How much detail should a lineage record have?

Enough to answer the questions you get asked: which sources and versions, which parameters changed the answer, which code and library versions, and when it ran. Full call-level provenance is rarely read.

Should I hash my input files?

Yes, as downloaded. That is what turns "we used the 2024 extract" into a checkable claim when the file at that path has since changed.

Why can I not just hash my outputs the same way?

Because some spatial formats are not byte-reproducible. Writing the same 15 wards twice to the same file names changed the GeoPackage's SHA-256, because it stores its own last-change time; GeoJSON, FlatGeobuf, the shapefile and Parquet came out identical — but GeoJSON and FlatGeobuf embed the file name as the layer name, and a shapefile's .dbf embeds the date, so those are stable only under the same name on the same day.

Do library versions really change the result?

Yes. GEOS, PROJ and GDAL have all changed the behaviour of common operations between releases, and an unexplained difference between two runs is usually one of them.

Where should the lineage record be stored?

Beside the artefact so it travels with it, and in a central store so you can query across runs — for example to find every output that used a source you have just found to be wrong.