🧭 Concept Metadata, licensing & provenance Pipelines & automation Metadata, Licensing & Provenance · step 3 of 20 Updated

Provenance and lineage explained for spatial workflows

Problem statement

The number in the report is 4,812. Six months later somebody asks where it came from, and the honest answer is: from a notebook that has been edited since, run against a file that has been replaced, using a library that has been upgraded twice. The map is still on the wall, and nobody can reproduce it.

Lineage is the ordered record of what went into a result and what was done to it. Provenance is the broader question it answers: can this output be traced, explained and reproduced? Every other metadata field can be recomputed from the data; lineage cannot, because it describes events that have already happened.

This guide sets out what a lineage record has to contain to be worth keeping, how much detail is enough, and why the checksum you record for a spatial file needs care.

Quick answer

Record, for every output: the inputs with their versions and content hashes, the operations with their parameters, the code version, the environment, and when it ran.

run = {
    "output": "flood_zones_2025.gpkg",
    "inputs": [
        {"name": "ea_lidar_dtm_2024", "version": "2024-06", "sha256": "9f3c…", "licence": "OGL-UK-3.0"},
        {"name": "ea_fluvial_model",  "version": "v7",      "sha256": "41ab…", "licence": "internal"},
    ],
    "steps": [
        {"op": "breach_depressions", "lib": "pysheds 0.4", "params": {}},
        {"op": "flowdir", "lib": "pysheds 0.4", "params": {"routing": "d8"}},
        {"op": "threshold", "params": {"return_period_years": 200}},
        {"op": "polygonise", "lib": "rasterio 1.5.1", "params": {"min_area_m2": 100}},
    ],
    "code": {"repo": "flood-pipeline", "commit": "3f9a1c2", "dirty": False},
    "environment": {"python": "3.14.0", "geopandas": "1.1.4", "gdal": "3.11.0"},
    "run_at": "2026-01-20T09:14:33Z",
    "output_sha256": "c2d1…",
}

Everything in that record is either machine-readable at run time or a constant. None of it depends on anyone remembering anything.

Flow from source datasets through parameterised operations to an output, with versions attached at each link.
A link without a version is a broken link — the chain only reaches as far as the last version you recorded.

Step-by-step solution

1. Decide what an "output" is

Lineage attaches to an artefact. If your pipeline writes six files, it writes six lineage records — or one record listing six outputs. What it must not do is describe "the pipeline" in general, because the question people ask is always about a specific file.

2. Identify inputs by content, not by path

/data/current/dtm.tif identifies nothing: the file at that path changes. Record a name, a version or date, and a content hash. The hash is what turns "we used the 2024 DTM" into a checkable claim.

3. Hash the content, not the container

A byte checksum of a GeoPackage is not reproducible: writing the same GeoDataFrame twice produces different bytes, because the container records its own timestamps. In a direct test, two writes of an identical 120-feature layer gave different SHA-256 digests for GeoPackage, GeoJSON and FlatGeobuf, and identical digests only for shapefile and Parquet. For inputs you did not create, a byte hash of the file as downloaded is fine and is the right thing; for outputs you create, hash the canonicalised data. How to checksum spatial datasets so you can prove they match has the method.

4. Record parameters that change the answer, not every argument

A parameter belongs in the lineage if a different value would produce a different output that somebody might question: the routing algorithm, the threshold, the resampling method, the CRS, the simplification tolerance, the random seed. Buffer sizes, yes. Chunk sizes and thread counts, no — they change speed, not results.

5. Pin the code and the environment

A commit hash and a dirty flag, plus the versions of the libraries that do the arithmetic. GDAL, PROJ, GEOS and the interpolation library all change results between versions; recording them is what makes "it was different last year" a diagnosable statement.

6. Write the record at the same moment as the data

A lineage record produced by a separate step can be written for a file that was never produced. Write it in the same function that writes the output, from the same variables.

7. Keep it with the artefact and in the log

Beside the file so it travels; in a run log so you can ask questions across runs, such as which outputs used a source you have just discovered was wrong.

8. Make it queryable

The payoff comes when someone reports a problem with an input. A lineage store lets you answer "which published outputs used this file?" in a query instead of a week.

Comparison grid of three levels of lineage detail against the questions each can answer.
Detail is not free; record the level that answers the questions you will actually be asked.

Code examples

Example 1 — capture the environment and the code version 

import subprocess, sys, importlib.metadata as md, datetime

def environment():
    def version(pkg):
        try:
            return md.version(pkg)
        except md.PackageNotFoundError:
            return None
    try:
        commit = subprocess.check_output(["git", "rev-parse", "--short", "HEAD"], text=True).strip()
        dirty = bool(subprocess.check_output(["git", "status", "--porcelain"], text=True).strip())
    except Exception:
        commit, dirty = None, None
    return {
        "python": sys.version.split()[0],
        "packages": {p: version(p) for p in
                     ("geopandas", "shapely", "pyproj", "rasterio", "pandas", "numpy")},
        "git": {"commit": commit, "dirty": dirty},
        "run_at": datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds"),
    }

Example 2 — a step recorder that cannot be forgotten 

import functools, json, pathlib

class Lineage:
    def __init__(self, output):
        self.record = {"output": str(output), "inputs": [], "steps": [],
                       "environment": environment()}

    def input(self, name, path, version=None, licence=None):
        self.record["inputs"].append({
            "name": name, "path": str(path), "version": version, "licence": licence,
            "sha256": sha256_of(path),
        })

    def step(self, op, **params):
        def wrap(fn):
            @functools.wraps(fn)
            def inner(*a, **kw):
                self.record["steps"].append({"op": op, "params": params})
                return fn(*a, **kw)
            return inner
        return wrap

    def write(self, path=None):
        path = pathlib.Path(path or (self.record["output"] + ".lineage.json"))
        path.write_text(json.dumps(self.record, indent=2))
        return path

lin = Lineage("flood_zones_2025.gpkg")
lin.input("ea_lidar_dtm", "dtm_2024.tif", version="2024-06", licence="OGL-UK-3.0")

@lin.step("polygonise", min_area_m2=100, simplify_m=1.0)
def polygonise(raster): ...

Attaching the record to the function that does the work is what stops the two drifting apart.

Example 3 — ask which outputs used a bad input 

import json, pathlib

def outputs_using(source_name, store="lineage/"):
    hits = []
    for f in pathlib.Path(store).glob("*.lineage.json"):
        rec = json.loads(f.read_text())
        if any(i["name"] == source_name for i in rec["inputs"]):
            hits.append({"output": rec["output"], "run_at": rec["environment"]["run_at"]})
    return sorted(hits, key=lambda h: h["run_at"], reverse=True)

for hit in outputs_using("ea_lidar_dtm"):
    print(hit)

This query is the reason to keep lineage in a directory rather than only beside each file.

Explanation

Why lineage cannot be reconstructed

Extent, CRS, schema and feature counts are properties of the data and can be recomputed at any time. Lineage describes events: which file existed at that path on that day, which version of GEOS was installed, what the threshold was before somebody changed it. Once those have passed, nothing in the data records them.

Why versions of libraries belong in the record

Geometry operations are not stable across versions. GEOS has changed the behaviour of buffer, simplify and overlay noding between releases; PROJ has changed default transformation pipelines as new grids shipped; interpolation libraries change their defaults. A result that differs from last year's is usually explained by one of those, and only if they were recorded.

Why the input hash matters more than the input path

The commonest cause of an unreproducible result is that the input changed underneath a stable path. A content hash converts "we used the current extract" into a statement that can be tested against the file you still have, and it is what lets you distinguish "the data changed" from "the code changed".

Why the detail level is a decision

Full provenance — every function call, every intermediate — is expensive to capture and almost never read. The useful level is the one that answers the questions you get: which sources, which versions, which parameters, which code. Record that reliably rather than recording everything unreliably.

Two panels contrasting recomputable metadata — extent, schema, counts, checksums — with lineage facts that are lost once the run is over.
Everything on the left can wait; nothing on the right can.

Edge cases or notes

  • A dirty working tree invalidates the commit hash. Record the flag.
  • Random seeds are parameters. Any stochastic step needs its seed in the record.
  • Manual steps must be recorded as manual. "Edited in QGIS by hand" is lineage.
  • Downloads need the date and the URL. Remote sources change without notice.
  • Record failures too. A run that produced nothing still explains a gap in a series.
  • Do not put credentials in the record. Record the endpoint, not the token.
  • Lineage grows. Rotate or index it; a directory of a million JSON files is not queryable.
  • Sign it if it matters. For regulated work, a signed record is worth the extra step.

FAQ

What is the difference between provenance and lineage?

Lineage is the record of inputs, operations and versions. Provenance is the property that record gives you: the ability to trace, explain and reproduce an output.

How much detail should a lineage record have?

Enough to answer the questions you get asked: which sources and versions, which parameters changed the answer, which code and library versions, and when it ran. Full call-level provenance is rarely read.

Should I hash my input files?

Yes, as downloaded. That is what turns "we used the 2024 extract" into a checkable claim when the file at that path has since changed.

Why can I not just hash my outputs the same way?

Because several spatial formats are not byte-reproducible. Writing the same layer twice produced different SHA-256 digests for GeoPackage, GeoJSON and FlatGeobuf in a direct test; only shapefile and Parquet were identical.

Do library versions really change the result?

Yes. GEOS, PROJ and GDAL have all changed the behaviour of common operations between releases, and an unexplained difference between two runs is usually one of them.

Where should the lineage record be stored?

Beside the artefact so it travels with it, and in a central store so you can query across runs — for example to find every output that used a source you have just found to be wrong.