🧭 Concept Metadata, licensing & provenance Pipelines & automation Metadata, Licensing & Provenance · step 4 Notebook By · Updated

Dataset versioning explained

On this page

Problem statement

boundaries.gpkg, boundaries_v2.gpkg, boundaries_final.gpkg, boundaries_FINAL_use_this.gpkg. Everyone has the folder, and everyone has had the conversation that starts "which one did you use?" and ends with nobody knowing.

Versioning a spatial dataset is harder than versioning code for three reasons: the files are too large for a diff to be meaningful, the same logical dataset changes for two different reasons (the world changed, or your processing changed), and consumers hold copies you cannot update. A version scheme has to distinguish those and survive being copied onto somebody's laptop.

This guide sets out what a version identifies, how to number it, and how to make the number travel with the file rather than only with the folder it came from.

Quick answer

Give every release an immutable identifier, put it inside the data, and never overwrite a published version:

version = {
    "dataset": "east-sussex-flood-zones",
    "version": "2026.1",                       # release, not a filename
    "content_date": "2025-11-01",              # what the world looked like
    "released": "2026-01-20",
    "supersedes": "2025.2",
    "changes": ["2024 LiDAR replaces 2019", "min mapped area 100 m²", "schema unchanged"],
    "content_sha256": "c2d1…",
}

The two rules that matter: a published version is never modified, and the version identifier is stored inside the dataset — in the GeoPackage metadata, the GeoTIFF tags or a column — so a copy on somebody's laptop still knows what it is.

The parts of a dataset version record. what a version saysversion 2026.1epoch . revisionimmutablecontent_date 2025-11-01the worldwhat it describesreleased 2026-01-20the filewhen it shippedsupersedes 2025.2the chainstill downloadablecontent_sha256 c2d1…the bytescheckableTwo dates, because the world and the pipeline change separately spatialworkflow.io
Two dates, because the world changing and the pipeline changing are different events.

Two reasons data changes, breaking changes, no in-place edits

1. Separate the two reasons a dataset changes

A content change means the world, or the survey of it, is different: new buildings, a new census, a better DEM. A processing change means your code produced a different answer from the same inputs. Users care about the first and are surprised by the second, so the version scheme has to make both visible.

A two-part number does this cleanly: 2026.1 where 2026 is the content epoch and 1 the processing revision within it.

2. Decide what breaks compatibility

A schema change — a renamed column, a changed unit, a new code list — breaks every consumer. Treat it as a major change, announce it, and if possible publish the old schema in parallel for a period. A new row does not break anything.

3. Never modify a published version

If 2026.1 is wrong, publish 2026.2 and mark 2026.1 as withdrawn. Editing a file in place means two people with the same version identifier hold different data, which is worse than being wrong.

4. Put the identifier inside the file

A filename is not a version, because filenames are changed by the people who download them. Every format that can hold dataset metadata should carry the version; the ones that cannot get a column.

import shutil
from osgeo import gdal

shutil.copy("releases/v5.1.2/countries.gpkg", "countries_download.gpkg")
ds = gdal.OpenEx("countries_download.gpkg", gdal.OF_UPDATE | gdal.OF_VECTOR)
ds.SetMetadataItem("VERSION", "v5.1.2")
ds.SetMetadataItem("CONTENT_DATE", PUBLISHED["v5.1.2"])
ds = None

Path("countries_download.gpkg").rename("world.gpkg")              # a user renames it
print(gdal.OpenEx("world.gpkg", gdal.OF_VECTOR).GetMetadata())
{'CONTENT_DATE': '2022-05-13', 'VERSION': 'v5.1.2'}

The file was renamed, and it still knows what it is. (The content date here is the day Natural Earth published 5.1.2.)

5. Publish a manifest that lists every version

One JSON file listing versions, dates, checksums and URLs lets a consumer check whether their copy is current without downloading anything.

6. Version the metadata with the data

A corrected licence or a rewritten abstract is a new release, because somebody may have made a decision on the old one.

7. Decide how long each version is kept

An archive that keeps every version forever is expensive; one that keeps only the latest makes every published result unreproducible. A common compromise is to keep every content epoch indefinitely and only the latest processing revision within superseded epochs.

8. Make the pipeline record which version it used

Everything above is wasted if the run does not record the identifier. How to record lineage automatically in a pipeline covers the mechanics.

Four versioning schemes compared on ordering, immutability and meaning. four schemes_final_v2date onlyepoch.revisionsorts correctly✕no✓yes✓yestravels in the file✕no✕no✓yesseparates the two causes✕no✕no✓yessurvives a rename✕no✕no✓yesA filename is renamed by whoever downloads it spatialworkflow.io
The scheme has to distinguish a change in the world from a change in your code.

A manifest consumers can check against

The data used on this page

A real release history: Natural Earth's 1:110m countries as published in three tagged releases of the project's repository — v4.1.0, v5.0.0 and v5.1.2 — fetched from GitHub by tag, so each is exactly what was released. The setup writes each to releases/<version>/countries.gpkg, the layout a small publisher might use, and defines the canonical_hash the examples call: the content hash from checksumming spatial datasets, so a copy re-saved in another format or with its rows reordered still matches.

import hashlib
import json
from pathlib import Path

import geopandas as gpd
import numpy as np
import pandas as pd
import shapely

RELEASES = ["v4.1.0", "v5.0.0", "v5.1.2"]
for tag in RELEASES:
    out = Path(f"releases/{tag}/countries.gpkg")
    if not out.exists():
        out.parent.mkdir(parents=True, exist_ok=True)
        gpd.read_file("https://raw.githubusercontent.com/nvkelso/natural-earth-vector/"
                      f"{tag}/geojson/ne_110m_admin_0_countries.geojson").to_file(out)

def canonical_hash(path, decimals=9):
    gdf = gpd.read_file(path)
    geom = shapely.force_2d(gdf.geometry.values)
    geom = np.array([shapely.multipolygons([g]) if g.geom_type == "Polygon" else g for g in geom])
    geom = shapely.transform(shapely.normalize(geom), lambda xy: np.round(xy, decimals))
    tab = pd.DataFrame(gdf.drop(columns=gdf.geometry.name)).astype(str)
    tab["_geometry"] = shapely.to_wkb(geom, hex=True)
    tab = tab.reindex(sorted(tab.columns), axis=1)
    tab = tab.sort_values(list(tab.columns)).reset_index(drop=True)
    return hashlib.sha256(tab.to_csv(index=False).encode()).hexdigest()

import requests
PUBLISHED = {tag: requests.get("https://api.github.com/repos/nvkelso/natural-earth-vector/releases/tags/" + tag,
                               timeout=60).json()["published_at"][:10] for tag in RELEASES}
Path("manifest.json").write_text(json.dumps({"dataset": "ne-110m-countries", "versions": []}))
print({tag: len(gpd.read_file(f"releases/{tag}/countries.gpkg")) for tag in RELEASES}, PUBLISHED)
{'v4.1.0': 177, 'v5.0.0': 177, 'v5.1.2': 177} {'v4.1.0': '2018-05-23', 'v5.0.0': '2021-12-08', 'v5.1.2': '2022-05-13'}

Example 1 — a manifest consumers can check against

import json, pathlib, datetime

def manifest_entry(path, dataset, version, content_date, supersedes=None, changes=()):
    return {
        "dataset": dataset,
        "version": version,
        "content_date": content_date,
        "released": datetime.date.today().isoformat(),
        "supersedes": supersedes,
        "changes": list(changes),
        "file": pathlib.Path(path).name,
        "bytes": pathlib.Path(path).stat().st_size,
        "content_sha256": canonical_hash(path),
        "url": f"https://data.example.org/{dataset}/{version}/{pathlib.Path(path).name}",
    }

manifest = json.loads(pathlib.Path("manifest.json").read_text())
history = [("v4.1.0", PUBLISHED["v4.1.0"], None, ["population estimates for 2017"]),
           ("v5.0.0", PUBLISHED["v5.0.0"], "v4.1.0", ["population estimates for 2019", "94 columns added, 4 removed"]),
           ("v5.1.2", PUBLISHED["v5.1.2"], "v5.0.0", ["point release"])]
for version, content_date, supersedes, changes in history:
    manifest["versions"].append(manifest_entry(
        f"releases/{version}/countries.gpkg", "ne-110m-countries", version, content_date,
        supersedes=supersedes, changes=changes))
pathlib.Path("manifest.json").write_text(json.dumps(manifest, indent=2))
for v in manifest["versions"]:
    print(v["version"], v["content_date"], f"{v['bytes']:,} bytes", v["content_sha256"][:12])
v4.1.0 2018-05-23 413,696 bytes f707b0837c27
v5.0.0 2021-12-08 548,864 bytes e19c96127589
v5.1.2 2022-05-13 557,056 bytes 216124f3b755

The content dates are when Natural Earth published each release, read from GitHub; released in a real manifest is the day you publish, which here is the day the example ran.

Example 2 — tell a consumer whether their copy is current

import requests

def check_version(local_path, manifest_url):
    manifest = requests.get(manifest_url, timeout=30).json()
    latest = max(manifest["versions"], key=lambda v: (v["content_date"], v["version"]))
    mine = canonical_hash(local_path)
    match = next((v for v in manifest["versions"] if v["content_sha256"] == mine), None)
    if match is None:
        return f"local copy matches no published version (latest is {latest['version']})"
    if match["version"] == latest["version"]:
        return f"up to date: {match['version']}"
    return (f"local copy is {match['version']}, latest is {latest['version']} "
            f"({latest['content_date']}): " + "; ".join(latest["changes"]))
import http.server, threading, functools

server = http.server.ThreadingHTTPServer(("127.0.0.1", 8765), functools.partial(
    http.server.SimpleHTTPRequestHandler, directory="."))
threading.Thread(target=server.serve_forever, daemon=True).start()
url = "http://127.0.0.1:8765/manifest.json"

gpd.read_file("releases/v5.0.0/countries.gpkg").to_file("my_copy.geojson")        # an old copy, re-saved
edited = gpd.read_file("releases/v5.1.2/countries.gpkg")
edited.loc[edited["NAME"] == "France", "POP_EST"] = 1                              # somebody's fix
edited.to_file("edited_copy.gpkg")
for copy in ("releases/v5.1.2/countries.gpkg", "my_copy.geojson", "edited_copy.gpkg"):
    print(f"{copy:34}", check_version(copy, url))
server.shutdown()
releases/v5.1.2/countries.gpkg     up to date: v5.1.2
my_copy.geojson                    local copy is v5.0.0, latest is v5.1.2 (2022-05-13): point release
edited_copy.gpkg                   local copy matches no published version (latest is v5.1.2)

The old copy is recognised although it is now a GeoJSON rather than the GeoPackage that was published — that is what a content hash buys over a file hash. The edited one matches nothing.

"Matches no published version" is the most useful answer this can give: it means somebody edited the file.

Example 3 — detect what actually changed between two versions

import geopandas as gpd, pandas as pd

def diff_versions(old_path, new_path, key):
    a = gpd.read_file(old_path).set_index(key)
    b = gpd.read_file(new_path).set_index(key)

    report = {
        "added": sorted(b.index.difference(a.index).tolist())[:10],
        "removed": sorted(a.index.difference(b.index).tolist())[:10],
        "n_added": len(b.index.difference(a.index)),
        "n_removed": len(a.index.difference(b.index)),
        "schema_added": sorted(set(b.columns) - set(a.columns)),
        "schema_removed": sorted(set(a.columns) - set(b.columns)),
    }
    common = a.index.intersection(b.index)
    moved = ~a.loc[common].geometry.geom_equals_exact(b.loc[common].geometry, tolerance=1e-6)
    report["n_geometry_changed"] = int(moved.sum())
    for col in set(a.columns) & set(b.columns) - {a.geometry.name}:
        changed = (a.loc[common, col].fillna("∅") != b.loc[common, col].fillna("∅")).sum()
        if changed:
            report.setdefault("attribute_changes", {})[col] = int(changed)
    return report
for old, new in (("v4.1.0", "v5.0.0"), ("v5.0.0", "v5.1.2")):
    report = diff_versions(f"releases/{old}/countries.gpkg", f"releases/{new}/countries.gpkg", "ADM0_A3")
    changed = sorted(report.get("attribute_changes", {}).items(), key=lambda kv: (-kv[1], kv[0]))[:4]
    print(f"{old} -> {new}: +{report['n_added']} -{report['n_removed']} features, "
          f"{report['n_geometry_changed']} geometries changed, "
          f"{len(report['schema_added'])} columns added, {len(report['schema_removed'])} removed")
    print("   most-changed attributes:", changed)
v4.1.0 -> v5.0.0: +0 -0 features, 120 geometries changed, 94 columns added, 4 removed
   most-changed attributes: [('POP_EST', 175), ('POP_YEAR', 175), ('GDP_YEAR', 166), ('POP_RANK', 14)]
v5.0.0 -> v5.1.2: +0 -0 features, 0 geometries changed, 8 columns added, 1 removed
   most-changed attributes: [('FCLASS_ISO', 175), ('MIN_LABEL', 23), ('NOTE_ADM0', 5), ('TYPE', 5)]

The second line is the case this function exists for. The change list above describes 5.1.2 as a point release — and the diff finds eight columns added, one removed and a changed FCLASS_ISO on 175 of the 177 countries. Nothing moved geographically, and a consumer joining on those columns would still find their code broken. A release note has to come from the diff, not from the version number.

Run it before publishing. A release whose change list says "minor update" and whose diff says 40% of geometries moved is a release that needs a different change list.

Why filenames fail as versions

Why filenames fail as versions

Filenames are renamed on download, truncated by email systems, and duplicated by "copy of". They also carry no ordering that a program can rely on — v10 sorts before v2. The version has to be inside the file, and the filename is at best a convenience.

Why the content date and the release date must both appear

A dataset published in 2026 describing 2024 is 2024 data, and every analysis using it should say so. Publishing only one date forces users to guess, and they guess the one that makes their analysis look current.

Why in-place edits are the real problem

Two people holding files both labelled 2026.1 that differ is unrecoverable: neither can tell, and any comparison between their results is meaningless. Immutability is the property that makes a version identifier mean anything at all, and it costs only storage.

Why a schema change deserves its own treatment

Adding rows breaks nothing. Renaming a column breaks every script, every style file and every join downstream, usually silently — a missing column becomes a null, and a null becomes a zero. Announce schema changes, and where the consumers are known, ship both shapes for a transition period.

Four kinds of dataset change and what each breaks downstream. what breaks a consumerchangeeffect downstreamrows added or removednothing breaks; counts movegeometry refinedareas and joins shift slightlycolumn renamed or retypedevery script breaks, usuallysilentlycode list extendedstyles and filters miss the newclassOnly the bottom two need announcing; both are schema changes spatialworkflow.io
A new row breaks nothing; a renamed column breaks everything, quietly.

Rolling datasets, derived products and large files

  • Rolling datasets need snapshots. A continuously updated source such as OpenStreetMap is versioned by the extract date.
  • Derived products inherit versions. Record the input version in the output's metadata.
  • Large files do not diff. Compare features and schema, not bytes.
  • Git LFS is rarely the answer. Object storage plus a manifest scales better for hundreds of megabytes.
  • Tile sets need cache-busting. A new version behind the same URL will be served from a cache for days.
  • Withdrawn versions should stay downloadable. Marked withdrawn, not deleted, or old results become unverifiable.
  • Dates in filenames are not immutability. They help humans and guarantee nothing.
  • A version is also a metadata field. It belongs in the record, not only in the manifest.

FAQ

How should I number spatial dataset versions?

Use two parts: a content epoch for when the data describes the world, and a revision for reprocessing within it — 2026.1. Semantic versioning works too, but only if you define what "breaking" means for your schema.

Where should the version identifier live?

Inside the dataset, in GeoPackage metadata, GeoTIFF tags or a column. A filename is renamed the moment somebody downloads it.

Can I just overwrite the file when I fix something?

No. Two people then hold different data under the same identifier, and nothing they compute can be compared. Publish a new version and mark the old one withdrawn.

Should I delete old versions?

Mark them withdrawn rather than deleting them, or every published result that used them becomes unverifiable. Keeping every content epoch and only the latest revision within superseded epochs is a common compromise.

How do I tell users what changed?

Publish a change list per version and, before releasing, run a feature-level diff to check the list matches reality.

Does Git work for spatial data?

For small vector files, yes. For anything large, use object storage with a manifest listing versions, checksums and URLs — Git LFS tends to be more trouble than the alternative.