How to checksum spatial datasets so you can prove they match

Problem statement

Two people want to be sure they have the same dataset. The obvious answer — hash the file — works for a download and is fragile for anything you produced yourself, because the bytes a spatial format writes depend on more than the data. Writing the UK's 361 local authority districts twice to the same path, a second and a half apart:

So a byte hash answers "is this the same file?", and it changes when the file is renamed, rewritten the next day or written as a GeoPackage at all. You usually want to ask "is this the same data?" — which needs a hash of the content, canonicalised so that the format's own choices do not change the answer.

Quick answer

Hash the data, not the container:

import hashlib, numpy as np, pandas as pd, shapely

def content_hash(gdf, decimals=9):
    """Stable across formats: geometry type, ring order, row order and column order."""
    geom = shapely.force_2d(gdf.geometry.values)
    geom = np.array([shapely.multipolygons([g]) if g.geom_type == "Polygon" else g
                     for g in geom])
    geom = shapely.transform(shapely.normalize(geom), lambda xy: np.round(xy, decimals))

    tab = pd.DataFrame(gdf.drop(columns=gdf.geometry.name)).astype(str)
    tab["_geometry"] = shapely.to_wkb(geom, hex=True)
    tab = tab.reindex(sorted(tab.columns), axis=1)
    tab = tab.sort_values(list(tab.columns)).reset_index(drop=True)
    return hashlib.sha256(tab.to_csv(index=False).encode()).hexdigest()

def read_any(path):
    return gpd.read_parquet(path) if str(path).endswith(".parquet") else gpd.read_file(path)

for d in (6, 9, 12):
    digests = {f.suffix: content_hash(read_any(f), d)[:12] for f in written}
    print(f"{d:2} decimals: {len(set(digests.values()))} distinct digest(s) {digests}")

 6 decimals: 1 distinct digest(s) {'.gpkg': 'c2561985ee5b', '.geojson': 'c2561985ee5b', '.fgb': 'c2561985ee5b', '.shp': 'c2561985ee5b', '.parquet': 'c2561985ee5b'}
 9 decimals: 1 distinct digest(s) {'.gpkg': '43e83380a88b', '.geojson': '43e83380a88b', '.fgb': '43e83380a88b', '.shp': '43e83380a88b', '.parquet': '43e83380a88b'}
12 decimals: 2 distinct digest(s) {'.gpkg': '3ac4e8cfb158', '.geojson': 'e8739ef4a895', '.fgb': '3ac4e8cfb158', '.shp': '3ac4e8cfb158', '.parquet': '3ac4e8cfb158'}

At six and nine decimal places the five formats give one digest. At twelve, GeoJSON's text round trip shows through.

The rounding is numeric on purpose. The first version of this function rounded by formatting WKT with rounding_precision=9, and GeoJSON still hashed differently at nine places: WKT writes numbers near zero in scientific notation with that many significant digits, and the UK straddles the Greenwich meridian — a longitude written as 3.691980419e-5 from one file and 3.69198042e-5 from another is a difference of 10⁻¹⁴ degrees, and a different digest.

Step-by-step solution

1. Use a byte hash for files you received

For a download, the byte hash is exactly right: it proves you have the same bytes the publisher shipped, and any difference is corruption or tampering. Record it with the URL and the date.

2. Use a content hash for data you produced

For an output, the byte hash is unstable and therefore useless as a claim. Hash the canonicalised content instead, and record the canonicalisation parameters with the digest.

3. Normalise geometry type

GeoPackage and FlatGeobuf declare a single layer geometry type, so both promote every Polygon to MultiPolygon on write. The districts are 231 Polygons and 130 MultiPolygons; both formats returned 361 MultiPolygons, while GeoJSON, shapefile and Parquet kept the mix. Wrapping every Polygon in a MultiPolygon before hashing removes the difference.

4. Normalise ring order and orientation

shapely.normalize puts rings and coordinate sequences into a canonical order, so two topologically identical geometries stored with different winding hash the same.

5. Normalise row order

FlatGeobuf reorders features by a packed Hilbert R-tree, so the first feature in the file is not the first feature you wrote — Hartlepool (E06000001) went in first and E07000062 came out first. Sorting the table before hashing makes the digest independent of storage order.

6. Choose a coordinate precision and state it

Text formats do not round-trip doubles exactly. A GeoJSON round trip left 307 of the 361 geometries bit-identical, with a maximum Hausdorff displacement of 9.9 × 10⁻¹⁴ degrees — about 10 nanometres, and enough to change a digest. Rounding to a stated precision when hashing absorbs it:

back = gpd.read_file("layer.geojson")
same = shapely.equals_exact(districts.geometry.values, back.geometry.values, tolerance=0)
print(f"{same.sum()} of {len(districts)} bit-identical; max displacement "
      f"{shapely.hausdorff_distance(districts.geometry.values, back.geometry.values).max():.1e}°")

307 of 361 bit-identical; max displacement 9.9e-14°

7. Or snap before writing, which is better

If you control the writer, snap coordinates to the precision the data deserves before you write:

snapped = districts.set_geometry(shapely.set_precision(districts.geometry.values, 1e-6, mode="pointwise"))
snapped.to_file("snapped.geojson")
again = gpd.read_file("snapped.geojson")
print(shapely.equals_exact(snapped.geometry.values, again.geometry.values, tolerance=0).sum(),
      "of", len(snapped), "bit-identical after the round trip")

361 of 361 bit-identical after the round trip

mode="pointwise" is needed on this layer. The default mode snaps and then repairs, and it raised TopologyException: side location conflict at -4.270698 51.676798: one of the 361 districts, Carmarthenshire, is an invalid polygon as published. Pointwise mode only rounds each vertex, which is what a precision snap before writing should do; repairing the geometry is a separate decision.

8. Record the digest with its recipe

A hash with no stated precision, normalisation and column handling cannot be reproduced by anyone else. Store the parameters next to the digest.

Code examples

The data used on this page

The UK's 361 local authority districts (May 2024, generalised) from the ONS Open Geography Portal, under the Open Government Licence. The setup writes them twice to each of five formats, at the same path each time, and compares the bytes; the table in the problem statement is this run.

import hashlib
import time
from pathlib import Path

import geopandas as gpd
import numpy as np
import shapely

districts = gpd.read_file(
    "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/"
    "Local_Authority_Districts_May_2024_Boundaries_UK_BGC/FeatureServer/0/query"
    "?where=1%3D1&outFields=LAD24CD,LAD24NM&outSR=4326&f=geojson")

def sha(path):
    return hashlib.sha256(Path(path).read_bytes()).hexdigest()

def shp_sha(path):                      # every part, keyed by suffix
    return hashlib.sha256("".join(sha(p) for p in sorted(Path(path).parent.glob(Path(path).stem + ".*"))).encode()).hexdigest()

written, results = [], {}
for ext, driver in (("gpkg", "GPKG"), ("geojson", "GeoJSON"), ("fgb", "FlatGeobuf"),
                    ("shp", "ESRI Shapefile"), ("parquet", None)):
    path = Path(f"layer.{ext}")
    digests = []
    for _ in range(2):
        districts.to_parquet(path) if driver is None else districts.to_file(path, driver=driver)
        digests.append(shp_sha(path) if ext == "shp" else sha(path))
        time.sleep(1.5)
    results[ext] = digests[0] == digests[1]
    written.append(path)
print(len(districts), districts.geom_type.value_counts().to_dict())
print("identical bytes on a rewrite:", results)
print(".dbf header date:", list(Path("layer.dbf").read_bytes()[1:4]), "(years since 1900, month, day)")
for f in ("layer.gpkg", "layer.fgb"):
    back = gpd.read_file(f)
    print(f, back.geom_type.value_counts().to_dict(), "first:", back["LAD24CD"].iloc[0])
for ext in ("geojson", "fgb"):
    districts.to_file(f"renamed_layer.{ext}")
    print(f"{ext}: same bytes under another name:", sha(f"renamed_layer.{ext}") == sha(f"layer.{ext}"))

361 {'Polygon': 231, 'MultiPolygon': 130}
identical bytes on a rewrite: {'gpkg': False, 'geojson': True, 'fgb': True, 'shp': True, 'parquet': True}
.dbf header date: [126, 9, 30] (years since 1900, month, day)
layer.gpkg {'MultiPolygon': 361} first: E06000001
layer.fgb {'MultiPolygon': 361} first: E07000062
geojson: same bytes under another name: False
fgb: same bytes under another name: False

Example 1 — the byte hash, done properly

import hashlib, pathlib

def file_hash(path, chunk=1 << 20):
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(chunk), b""):
            h.update(block)
    return h.hexdigest()

def dataset_hash(path):
    """Shapefiles are several files; hash them together, in a fixed order."""
    p = pathlib.Path(path)
    if p.suffix.lower() == ".shp":
        sidecars = {".shp", ".shx", ".dbf", ".prj", ".cpg", ".qix", ".sbn", ".sbx"}
        parts = sorted(q for q in p.parent.glob(p.stem + ".*") if q.suffix.lower() in sidecars)
        h = hashlib.sha256()
        for part in parts:
            h.update(part.suffix.lower().encode())       # the suffix, not the name
            h.update(file_hash(part).encode())
        return h.hexdigest()
    return file_hash(p)

import shutil
for part in Path(".").glob("layer.*"):
    if part.suffix in (".shp", ".shx", ".dbf", ".prj", ".cpg"):
        shutil.copy(part, f"renamed{part.suffix}")
print("renamed copy, same digest:", dataset_hash("layer.shp") == dataset_hash("renamed.shp"))
Path("renamed.prj").unlink()
print("without its .prj, same digest:", dataset_hash("layer.shp") == dataset_hash("renamed.shp"))

renamed copy, same digest: True
without its .prj, same digest: False

Hashing each sidecar's suffix with its contents is what stops a missing .prj from going unnoticed. The first version had two faults, both found by this run. It hashed each part's full name, so a byte-identical copy called anything other than layer hashed differently. And it took every file whose name started layer. — which in a working folder included layer.gpkg, layer.geojson and layer.parquet, none of them part of the shapefile.

Example 2 — prove two files hold the same data

import geopandas as gpd

def same_data(a, b, decimals=9, rename_truncated=False):
    ga = gpd.read_parquet(a) if str(a).endswith(".parquet") else gpd.read_file(a)
    gb = gpd.read_parquet(b) if str(b).endswith(".parquet") else gpd.read_file(b)
    if rename_truncated:                       # shapefile truncates names to 10 characters
        gb = gb.rename(columns={c[:10]: c for c in ga.columns if len(c) > 10})
    return content_hash(ga, decimals) == content_hash(gb, decimals)

wide = districts.assign(local_authority_name=districts["LAD24NM"])
wide.to_parquet("wide.parquet"); wide.to_file("wide.shp")
print(list(gpd.read_file("wide.shp").columns))
print("same data:", same_data("wide.parquet", "wide.shp"),
      "| with rename_truncated:", same_data("wide.parquet", "wide.shp", rename_truncated=True))

['LAD24CD', 'LAD24NM', 'local_auth', 'geometry']
same data: False | with rename_truncated: True

The rename is by name, not position. The first version assigned gb.columns = list(ga.columns), which assumes both files list their columns in the same order; the shapefile put local_auth before the geometry and the Parquet file after it, so the geometry column was renamed to local_authority_name and hashing failed with a TypeError.

The rename_truncated flag is not a hack to hide a difference — it is the explicit acknowledgement that a shapefile cannot carry a column called local_authority_name, which it stores as local_auth.

Example 3 — a raster content hash

import rasterio, hashlib, numpy as np

def raster_content_hash(path, round_to=None):
    with rasterio.open(path) as src:
        h = hashlib.sha256()
        h.update(str(src.crs).encode())
        h.update(np.array(src.transform[:6], dtype="float64").round(9).tobytes())
        h.update(f"{src.width}x{src.height}x{src.count}".encode())
        for i in range(1, src.count + 1):
            band = src.read(i)
            if round_to is not None and np.issubdtype(band.dtype, np.floating):
                band = np.round(band, round_to)
            nodata = src.nodatavals[i - 1]
            if nodata is not None and np.isnan(nodata):
                band = np.where(np.isnan(band), np.float64(-1e308), band)
            h.update(np.ascontiguousarray(band).tobytes())
        return h.hexdigest()

A real raster to try it on — the Copernicus GLO-90 elevation of the Isle of Wight, with the sea set to NaN — written twice with the same values, once with NaN from np.nan and once with the NaN a float operation produces:

import rasterio
from osgeo import gdal

gdal.Warp("dem.tif", "/vsicurl/https://copernicus-dem-90m.s3.amazonaws.com/Copernicus_DSM_COG_30_N50_00_W002_00_DEM/"
          "Copernicus_DSM_COG_30_N50_00_W002_00_DEM.tif", outputBounds=(-1.6, 50.57, -1.06, 50.77))
with rasterio.open("dem.tif") as src:
    dem, profile = src.read(1).astype("float32"), src.profile | {"dtype": "float32", "nodata": np.nan}
sea = dem <= 0
with np.errstate(invalid="ignore"):
    for name, nan in (("dem_a.tif", np.float32(np.nan)), ("dem_b.tif", np.float32(0) / np.float32(0))):
        with rasterio.open(name, "w", **profile) as dst:
            dst.write(np.where(sea, nan, dem), 1)
print("NaN bytes:", np.float32(np.nan).tobytes().hex(), "vs",
      (np.float32(0) / np.float32(0)).tobytes().hex() if np.seterr(invalid="ignore") else "")
print("pixels read back identical:", np.array_equal(rasterio.open("dem_a.tif").read(1),
                                                   rasterio.open("dem_b.tif").read(1), equal_nan=True))
print("file hashes equal:   ", file_hash("dem_a.tif") == file_hash("dem_b.tif"))
print("content hashes equal:", raster_content_hash("dem_a.tif") == raster_content_hash("dem_b.tif"))

NaN bytes: 0000c07f vs 0000c0ff
pixels read back identical: True
file hashes equal:    False
content hashes equal: True

The NaN substitution matters because NaN is not one bit pattern. np.nan is 0x7fc00000 as a float32; the NaN that 0/0 produces on x86 has its sign bit set, 0xffc00000. Both are NaN, both mean "no data", and they hash differently — so two rasters that compare equal value for value can fail a byte-level check. Replacing every NaN with one sentinel before hashing makes the comparison say what you mean.

Explanation

Why GeoPackage and GeoJSON are not byte-reproducible

A GeoPackage records its own last-change timestamp in gpkg_contents, so two writes a second apart differ. GeoJSON and FlatGeobuf carry no timestamp, but they carry the layer name, which GDAL takes from the file name — the same features written as a.geojson and b.geojson differ in one line. A shapefile's .dbf header holds the date it was written, so a rewrite tomorrow differs from one today. Parquet, in this test, carried nothing that varied — which makes it the one format where a byte hash is a dependable output check.

Why the format's normalisation is the interesting part

Every difference on this page is a format doing something reasonable: declaring one geometry type per layer, ordering features spatially for fast reads, writing coordinates as decimal text. None is a bug, and all of them change the bytes. Canonicalising before hashing is how you compare the data through those choices.

Why coordinate precision has to be a decision

Doubles have about 15–17 significant decimal digits, and a text round trip loses the last one or two. At UK latitudes a difference of 10⁻¹³ degrees is about 10 nanometres — physically meaningless and digest-changing. Choosing a precision is choosing how much agreement you require, and nine decimal places is about 0.1 mm, which is beyond any survey.

Why snapping before writing is the better fix

Rounding at hash time hides the difference; snapping at write time removes it. A dataset whose coordinates have been snapped to a stated grid round-trips exactly through every format, hashes identically everywhere, and is honest about its own precision. The cost is one line in the writer.

Edge cases or notes

• Hash the schema too if a column rename matters to you.
• Column order is storage. Sort columns before hashing, or the digest changes when someone reorders them.
• Empty and null geometries differ. Decide which you treat as equal.
• Z and M values are dropped by force_2d; keep them if they are data.
• Categorical dtypes stringify differently. Casting the whole table to str avoids surprises.
• Large datasets need a streaming hash. Hash per chunk and combine, in a fixed chunk order.
• SHA-256 is right here. MD5 is for detecting corruption, not for claims.
• Record the recipe with the digest. A hash without its parameters is not reproducible.

Internal links

• Provenance and lineage explained for spatial workflows — where the digest is recorded
• Dataset versioning explained — checksums as version identity
• How to write a metadata record for a dataset in Python — the field the digest goes in
• Metadata disappears when you convert the file — the other side of format differences
• You cannot tell which data version produced a map — what the digest answers
• How to validate pipeline inputs and outputs in Python — hashing as a pipeline check
• Coordinate precision explained — choosing the rounding
• GeoParquet and columnar storage explained — one of the two reproducible formats

FAQ

Why does hashing a GeoPackage give a different result each time?

Because the container records its own last-change timestamp. Two writes of the same layer to the same path a second and a half apart produced different SHA-256 digests; GeoJSON, FlatGeobuf, shapefile and Parquet were identical — as long as the file name and, for a shapefile, the date did not change.

How do I prove two spatial files hold the same data?

Hash the canonicalised content: normalise geometry type and ring order, sort the rows and columns, round coordinates to a stated precision, then hash the result.

Which formats are byte-reproducible?

Written twice to the same path, Parquet, GeoJSON, FlatGeobuf and shapefile were; GeoPackage was not. GeoJSON and FlatGeobuf change with the file name and a shapefile with the date, so only Parquet was reproducible without conditions.

What coordinate precision should I hash at?

Nine decimal degrees is about 0.1 mm and safely below any survey precision. State whatever you choose alongside the digest.

Why did my GeoJSON hash differ from the source?

Text formats do not round-trip doubles exactly — a round trip left 307 of 361 district geometries bit-identical, with a maximum displacement of about 10 nanometres. Snap the coordinates before writing, or round numerically when hashing.

Should I use MD5 or SHA-256?

SHA-256. MD5 is adequate for detecting accidental corruption and inadequate for anything you want to assert.