🔧 Fix Metadata, licensing & provenance Pipelines & automation Metadata, Licensing & Provenance · step 17 of 20 Updated

You cannot tell which data version produced a map

Problem statement

A figure in a report shows 4,812 properties at risk. A rerun of the same notebook gives 4,790. Nobody changed the code — or nobody thinks they did — and there is no way to tell whether the source data was updated, a library changed a default, or a parameter was edited and committed as "tidy up".

This is the failure that every other guide in this cluster exists to prevent, and by the time you are asking the question the information has usually gone. This guide covers what can still be recovered, and how to make the next version answerable in a minute rather than a week.

Quick answer

Work backwards through what is still on disk:

import subprocess, pathlib, datetime, geopandas as gpd

# 1. What does the artefact itself say?
print(gpd.read_file("flood_zones.gpkg").crs, len(gpd.read_file("flood_zones.gpkg")))
from osgeo import gdal
print(gdal.OpenEx("flood_zones.gpkg").GetMetadata())          # VERSION? CONTENT_DATE?

# 2. When was the figure made, and what did the repository look like then?
made = datetime.datetime.fromtimestamp(pathlib.Path("figure_3.png").stat().st_mtime)
print(subprocess.run(["git", "log", "-1", "--before", made.isoformat(), "--oneline"],
                     capture_output=True, text=True).stdout)

# 3. Which inputs existed then, and have they changed since?
for p in pathlib.Path("data").glob("*"):
    print(p.name, datetime.datetime.fromtimestamp(p.stat().st_mtime).date(), p.stat().st_size)

File modification times and git log --before will usually narrow it to a day and a commit. That is enough to test a hypothesis; it is not enough to prove one.

Triage of causes for an unreproducible figure and the evidence that identifies each.
Four causes, four different pieces of evidence — and only the last one leaves no trace at all.

Step-by-step solution

1. Check the artefact for an embedded version

A GeoPackage may have a VERSION item, a GeoTIFF may have tags, a CSV may have a header comment. It is worth thirty seconds before anything else.

2. Compare the output against a rerun, feature by feature

If the difference is in a handful of features, it is a data change. If it is everywhere and small, it is a parameter or a library. If the schema changed, it is an upstream release.

a, b = gpd.read_file("old_output.gpkg"), gpd.read_file("new_output.gpkg")
print(len(a), len(b), sorted(set(a.columns) ^ set(b.columns)))

3. Test the data hypothesis

Re-download the input at the version you think was used, or find an archived copy, and run the current code against it. If the old number comes back, the data changed.

4. Test the environment hypothesis

Recreate the environment at the date of the figure from the lock file, if one exists, and rerun. GEOS, PROJ and GDAL all change behaviour between releases, and a changed default in a geometry operation produces exactly this pattern of small differences everywhere.

5. Test the code hypothesis

git log --since on the pipeline directory, looking for parameter changes rather than refactors. A diff restricted to numeric literals finds these fast:

git log -p --since=2025-09-01 -- pipeline/ | grep -E '^[-+].*[0-9]+\.?[0-9]*' | head -50

6. Accept that it may be unanswerable, and fix the next one

If none of the above resolves it, say so in the report rather than guessing. Then make the next version answerable.

7. Put the version inside everything you publish from now on

The dataset, the figure, the report. A figure with a caption that ends flood_zones 2026.1 · pipeline 3f9a1c2 is unambiguous forever, and costs one f-string.

8. Start recording lineage

How to record lineage automatically in a pipeline is the permanent fix, and it is about fifty lines.

Panels contrasting an unstamped figure with one carrying dataset version, commit and run date.
The caption is the cheapest provenance mechanism there is.

Code examples

Example 1 — stamp every figure 

import matplotlib.pyplot as plt, subprocess, datetime

def provenance_stamp(dataset_version, extra=""):
    try:
        commit = subprocess.check_output(["git", "rev-parse", "--short", "HEAD"],
                                         text=True, stderr=subprocess.DEVNULL).strip()
        dirty = "+" if subprocess.check_output(["git", "status", "--porcelain"],
                                               text=True, stderr=subprocess.DEVNULL).strip() else ""
    except Exception:
        commit, dirty = "nogit", ""
    date = datetime.date.today().isoformat()
    return f"{dataset_version} · {commit}{dirty} · {date}{(' · ' + extra) if extra else ''}"

fig, ax = plt.subplots()
ax.set_title("Properties at risk, 1-in-200-year extent")
fig.text(0.99, 0.005, provenance_stamp("flood_zones 2026.1"),
         ha="right", va="bottom", fontsize=5.5, color="#666666")
fig.savefig("figure_3.png", dpi=200)

The + for a dirty tree is the detail that matters: a commit hash from an uncommitted working tree identifies nothing, and marking it is more honest than omitting it.

Example 2 — stamp the numbers, not only the pictures 

import json, pathlib

def write_result(name, value, dataset_version, **context):
    record = {"name": name, "value": value, "dataset_version": dataset_version,
              "stamp": provenance_stamp(dataset_version), **context}
    path = pathlib.Path("results") / f"{name}.json"
    path.parent.mkdir(exist_ok=True)
    path.write_text(json.dumps(record, indent=2))
    return value

at_risk = write_result("properties_at_risk", 4812, "flood_zones 2026.1",
                       return_period=200, threshold_m=0.3)

A report generated from these files cannot quote a number without its provenance, because the number and the stamp come out of the same object.

Example 3 — find the candidate commits for a figure 

import subprocess, pathlib, datetime

def commits_around(figure_path, pipeline_dir="pipeline", days=3):
    made = datetime.datetime.fromtimestamp(pathlib.Path(figure_path).stat().st_mtime)
    since = (made - datetime.timedelta(days=days)).isoformat()
    until = (made + datetime.timedelta(days=1)).isoformat()
    out = subprocess.run(
        ["git", "log", "--since", since, "--until", until, "--oneline", "--", pipeline_dir],
        capture_output=True, text=True).stdout
    return {"figure_made": made.isoformat(timespec="seconds"), "commits": out.splitlines()}

Modification times are weak evidence — a copy resets them — but they are usually the only evidence left, and they narrow the search to a handful of commits.

Explanation

Why "nobody changed the code" is usually true and irrelevant

Three things feed a result and only one of them is your code. The input file at a stable path is replaced on a schedule you do not control; the environment is rebuilt whenever a container is; and the parameters may live in a config file that is not reviewed as carefully as the code. The common case is a data refresh, and the reason it is hard to confirm is that nothing recorded which version was read.

Why library versions produce this exact symptom

A change in GEOS noding, a new default in a PROJ transformation pipeline, or a different resampling default changes many features by a small amount. That signature — everything moves slightly, nothing moves much — is almost diagnostic, and the test is to pin the old versions and rerun.

Why modification times are weak but worth using

mtime is set by the last write, and a copy, a sync or a restore resets it. It is still the most common surviving evidence, and combined with git log --before it usually narrows the search enough to test a hypothesis directly.

Why the caption is the highest-value fix

Everything else in this cluster requires infrastructure. A provenance stamp in a figure's corner requires one function and appears on every artefact that leaves the building. It does not tell you why a number changed, but it tells you exactly which inputs and which code produced it, which is the question that is otherwise unanswerable.

Vertical steps for recovering which data version produced a figure: artefact metadata, output diff, modification times with git log, pinning the old environment, and admitting it is unknown.
Five steps, in the order that costs least — the last one is a legitimate answer.

Edge cases or notes

  • Notebook outputs are evidence. A committed .ipynb records numbers and sometimes versions.
  • pip freeze in the container image is the environment record if nothing else exists.
  • Cloud storage keeps old versions. Object versioning may still hold the input you need.
  • Check the config, not only the code. A YAML parameter change is a code change.
  • Compare bounding boxes. An input whose extent grew is a re-release.
  • Check the CRS. A silently changed CRS produces plausible, wrong numbers.
  • Figure metadata can hold the stamp. PNG text chunks and PDF metadata both work.
  • Say "unknown" in the report. A guessed provenance is worse than an acknowledged gap.

FAQ

How do I find out which data version produced an old map?

Check the artefact for an embedded version, then use the figure's modification time with git log --before to find the code, then compare the current inputs' dates and sizes against that. It narrows it to a day; it rarely proves it.

The numbers changed and the code did not. What happened?

Usually the input data was refreshed at a stable path, or the environment was rebuilt with newer versions of GEOS, PROJ or GDAL. Both produce small differences across many features.

How can I tell a data change from a library change?

A data change shows as added, removed or altered features. A library change shows as many features differing by a small amount with the feature set unchanged.

What is the cheapest thing I can do today?

Stamp every figure and every published number with the dataset version, the commit hash and the run date. One function, and it appears on everything.

Do file modification times prove anything?

No. They are reset by copies, syncs and restores. They are useful for narrowing a search and not for concluding one.

What if I genuinely cannot reconstruct it?

Say so in the report. An acknowledged gap is defensible; a guess presented as provenance is not.