🧭 Concept Metadata, licensing & provenance Pipelines & automation Metadata, Licensing & Provenance · step 4 of 20 Updated

Dataset versioning explained

Problem statement

boundaries.gpkg, boundaries_v2.gpkg, boundaries_final.gpkg, boundaries_FINAL_use_this.gpkg. Everyone has the folder, and everyone has had the conversation that starts "which one did you use?" and ends with nobody knowing.

Versioning a spatial dataset is harder than versioning code for three reasons: the files are too large for a diff to be meaningful, the same logical dataset changes for two different reasons (the world changed, or your processing changed), and consumers hold copies you cannot update. A version scheme has to distinguish those and survive being copied onto somebody's laptop.

This guide sets out what a version identifies, how to number it, and how to make the number travel with the file rather than only with the folder it came from.

Quick answer

Give every release an immutable identifier, put it inside the data, and never overwrite a published version:

version = {
    "dataset": "east-sussex-flood-zones",
    "version": "2026.1",                       # release, not a filename
    "content_date": "2025-11-01",              # what the world looked like
    "released": "2026-01-20",
    "supersedes": "2025.2",
    "changes": ["2024 LiDAR replaces 2019", "min mapped area 100 m²", "schema unchanged"],
    "content_sha256": "c2d1…",
}

The two rules that matter: a published version is never modified, and the version identifier is stored inside the dataset — in the GeoPackage metadata, the GeoTIFF tags or a column — so a copy on somebody's laptop still knows what it is.

Stack showing a dataset version identifier, content date, release date, supersedes link and change list.
Two dates, because the world changing and the pipeline changing are different events.

Step-by-step solution

1. Separate the two reasons a dataset changes

A content change means the world, or the survey of it, is different: new buildings, a new census, a better DEM. A processing change means your code produced a different answer from the same inputs. Users care about the first and are surprised by the second, so the version scheme has to make both visible.

A two-part number does this cleanly: 2026.1 where 2026 is the content epoch and 1 the processing revision within it.

2. Decide what breaks compatibility

A schema change — a renamed column, a changed unit, a new code list — breaks every consumer. Treat it as a major change, announce it, and if possible publish the old schema in parallel for a period. A new row does not break anything.

3. Never modify a published version

If 2026.1 is wrong, publish 2026.2 and mark 2026.1 as withdrawn. Editing a file in place means two people with the same version identifier hold different data, which is worse than being wrong.

4. Put the identifier inside the file

A filename is not a version, because filenames are changed by the people who download them. Every format that can hold dataset metadata should carry the version; the ones that cannot get a column.

from osgeo import gdal
ds = gdal.OpenEx("flood_zones.gpkg", gdal.OF_UPDATE | gdal.OF_VECTOR)
ds.SetMetadataItem("VERSION", "2026.1")
ds.SetMetadataItem("CONTENT_DATE", "2025-11-01")
ds = None

5. Publish a manifest that lists every version

One JSON file listing versions, dates, checksums and URLs lets a consumer check whether their copy is current without downloading anything.

6. Version the metadata with the data

A corrected licence or a rewritten abstract is a new release, because somebody may have made a decision on the old one.

7. Decide how long each version is kept

An archive that keeps every version forever is expensive; one that keeps only the latest makes every published result unreproducible. A common compromise is to keep every content epoch indefinitely and only the latest processing revision within superseded epochs.

8. Make the pipeline record which version it used

Everything above is wasted if the run does not record the identifier. How to record lineage automatically in a pipeline covers the mechanics.

Comparison grid of filename suffixes, dates, semantic versions and content-epoch versions.
The scheme has to distinguish a change in the world from a change in your code.

Code examples

Example 1 — a manifest consumers can check against 

import json, pathlib, datetime

def manifest_entry(path, dataset, version, content_date, supersedes=None, changes=()):
    return {
        "dataset": dataset,
        "version": version,
        "content_date": content_date,
        "released": datetime.date.today().isoformat(),
        "supersedes": supersedes,
        "changes": list(changes),
        "file": pathlib.Path(path).name,
        "bytes": pathlib.Path(path).stat().st_size,
        "content_sha256": canonical_hash(path),
        "url": f"https://data.example.org/{dataset}/{version}/{pathlib.Path(path).name}",
    }

manifest = json.loads(pathlib.Path("manifest.json").read_text())
manifest["versions"].append(manifest_entry(
    "flood_zones.gpkg", "east-sussex-flood-zones", "2026.1", "2025-11-01",
    supersedes="2025.2", changes=["2024 LiDAR replaces 2019", "min mapped area 100 m²"]))
pathlib.Path("manifest.json").write_text(json.dumps(manifest, indent=2))

Example 2 — tell a consumer whether their copy is current 

import requests

def check_version(local_path, manifest_url):
    manifest = requests.get(manifest_url, timeout=30).json()
    latest = max(manifest["versions"], key=lambda v: (v["content_date"], v["version"]))
    mine = canonical_hash(local_path)
    match = next((v for v in manifest["versions"] if v["content_sha256"] == mine), None)
    if match is None:
        return f"local copy matches no published version (latest is {latest['version']})"
    if match["version"] == latest["version"]:
        return f"up to date: {match['version']}"
    return (f"local copy is {match['version']}, latest is {latest['version']} "
            f"({latest['content_date']}): " + "; ".join(latest["changes"]))

"Matches no published version" is the most useful answer this can give: it means somebody edited the file.

Example 3 — detect what actually changed between two versions 

import geopandas as gpd, pandas as pd

def diff_versions(old_path, new_path, key):
    a = gpd.read_file(old_path).set_index(key)
    b = gpd.read_file(new_path).set_index(key)

    report = {
        "added": sorted(b.index.difference(a.index).tolist())[:10],
        "removed": sorted(a.index.difference(b.index).tolist())[:10],
        "n_added": len(b.index.difference(a.index)),
        "n_removed": len(a.index.difference(b.index)),
        "schema_added": sorted(set(b.columns) - set(a.columns)),
        "schema_removed": sorted(set(a.columns) - set(b.columns)),
    }
    common = a.index.intersection(b.index)
    moved = ~a.loc[common].geometry.geom_equals_exact(b.loc[common].geometry, tolerance=1e-6)
    report["n_geometry_changed"] = int(moved.sum())
    for col in set(a.columns) & set(b.columns) - {a.geometry.name}:
        changed = (a.loc[common, col].fillna("∅") != b.loc[common, col].fillna("∅")).sum()
        if changed:
            report.setdefault("attribute_changes", {})[col] = int(changed)
    return report

Run it before publishing. A release whose change list says "minor update" and whose diff says 40% of geometries moved is a release that needs a different change list.

Explanation

Why filenames fail as versions

Filenames are renamed on download, truncated by email systems, and duplicated by "copy of". They also carry no ordering that a program can rely on — v10 sorts before v2. The version has to be inside the file, and the filename is at best a convenience.

Why the content date and the release date must both appear

A dataset published in 2026 describing 2024 is 2024 data, and every analysis using it should say so. Publishing only one date forces users to guess, and they guess the one that makes their analysis look current.

Why in-place edits are the real problem

Two people holding files both labelled 2026.1 that differ is unrecoverable: neither can tell, and any comparison between their results is meaningless. Immutability is the property that makes a version identifier mean anything at all, and it costs only storage.

Why a schema change deserves its own treatment

Adding rows breaks nothing. Renaming a column breaks every script, every style file and every join downstream, usually silently — a missing column becomes a null, and a null becomes a zero. Announce schema changes, and where the consumers are known, ship both shapes for a transition period.

Triage of four kinds of dataset change — rows, geometry, schema and code lists — against what each breaks for a consumer.
A new row breaks nothing; a renamed column breaks everything, quietly.

Edge cases or notes

  • Rolling datasets need snapshots. A continuously updated source such as OpenStreetMap is versioned by the extract date.
  • Derived products inherit versions. Record the input version in the output's metadata.
  • Large files do not diff. Compare features and schema, not bytes.
  • Git LFS is rarely the answer. Object storage plus a manifest scales better for hundreds of megabytes.
  • Tile sets need cache-busting. A new version behind the same URL will be served from a cache for days.
  • Withdrawn versions should stay downloadable. Marked withdrawn, not deleted, or old results become unverifiable.
  • Dates in filenames are not immutability. They help humans and guarantee nothing.
  • A version is also a metadata field. It belongs in the record, not only in the manifest.

FAQ

How should I number spatial dataset versions?

Use two parts: a content epoch for when the data describes the world, and a revision for reprocessing within it — 2026.1. Semantic versioning works too, but only if you define what "breaking" means for your schema.

Where should the version identifier live?

Inside the dataset, in GeoPackage metadata, GeoTIFF tags or a column. A filename is renamed the moment somebody downloads it.

Can I just overwrite the file when I fix something?

No. Two people then hold different data under the same identifier, and nothing they compute can be compared. Publish a new version and mark the old one withdrawn.

Should I delete old versions?

Mark them withdrawn rather than deleting them, or every published result that used them becomes unverifiable. Keeping every content epoch and only the latest revision within superseded epochs is a common compromise.

How do I tell users what changed?

Publish a change list per version and, before releasing, run a feature-level diff to check the list matches reality.

Does Git work for spatial data?

For small vector files, yes. For anything large, use object storage with a manifest listing versions, checksums and URLs — Git LFS tends to be more trouble than the alternative.