How to checksum spatial datasets so you can prove they match #

Problem statement #

Two people want to be sure they have the same dataset. The obvious answer — hash the file — works for a download and fails for anything you produced yourself, because several spatial formats are not byte-reproducible. Writing the same 120-feature layer twice, one second apart, gave different SHA-256 digests:

The three that differ embed timestamps or generator strings. So a byte hash answers "is this the same file?" and you usually want to ask "is this the same data?" — which needs a hash of the content, canonicalised so that the format's own choices do not change the answer.

Quick answer #

Hash the data, not the container:

import hashlib, numpy as np, pandas as pd, shapely

def content_hash(gdf, decimals=9):
    """Stable across formats: geometry type, ring order, row order and column order."""
    geom = shapely.force_2d(gdf.geometry.values)
    geom = np.array([shapely.multipolygons([g]) if g.geom_type == "Polygon" else g
                     for g in geom])
    geom = shapely.normalize(geom)

    tab = pd.DataFrame(gdf.drop(columns=gdf.geometry.name)).astype(str)
    tab["_geometry"] = shapely.to_wkt(geom, rounding_precision=decimals)
    tab = tab.reindex(sorted(tab.columns), axis=1)
    tab = tab.sort_values(list(tab.columns)).reset_index(drop=True)
    return hashlib.sha256(tab.to_csv(index=False).encode()).hexdigest()

Writing the same layer to GeoPackage, GeoJSON, FlatGeobuf, shapefile and Parquet and hashing each after reading it back gave one digest for all five at nine decimal places.

Step-by-step solution #

1. Use a byte hash for files you received #

For a download, the byte hash is exactly right: it proves you have the same bytes the publisher shipped, and any difference is corruption or tampering. Record it with the URL and the date.

2. Use a content hash for data you produced #

For an output, the byte hash is unstable and therefore useless as a claim. Hash the canonicalised content instead, and record the canonicalisation parameters with the digest.

3. Normalise geometry type #

GeoPackage and FlatGeobuf declare a single layer geometry type, so both promote every Polygon to MultiPolygon on write. In the test dataset — 64 MultiPolygons and 56 Polygons — both formats returned 120 MultiPolygons. Wrapping every Polygon in a MultiPolygon before hashing removes the difference.

4. Normalise ring order and orientation #

shapely.normalize puts rings and coordinate sequences into a canonical order, so two topologically identical geometries stored with different winding hash the same.

5. Normalise row order #

FlatGeobuf reorders features by a packed Hilbert R-tree, so the first feature in the file is not the first feature you wrote. Sorting the table before hashing makes the digest independent of storage order.

6. Choose a coordinate precision and state it #

Text formats do not round-trip doubles exactly. A GeoJSON round trip left only 4 of 120 geometries bit-identical, with a maximum Hausdorff displacement of 9.8 × 10⁻¹⁴ degrees — about 0.01 micrometres, and enough to change a digest. Rounding to a stated precision when hashing absorbs it.

7. Or snap before writing, which is better #

If you control the writer, snap coordinates to the precision the data deserves before you write. The same 120 features snapped to 10⁻⁶ degrees and round-tripped through GeoJSON came back with all 120 geometries bit-identical.

8. Record the digest with its recipe #

A hash with no stated precision, normalisation and column handling cannot be reproduced by anyone else. Store the parameters next to the digest.

Code examples #

Example 1 — the byte hash, done properly #

import hashlib, pathlib

def file_hash(path, chunk=1 << 20):
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(chunk), b""):
            h.update(block)
    return h.hexdigest()

def dataset_hash(path):
    """Shapefiles are several files; hash them together, in a fixed order."""
    p = pathlib.Path(path)
    if p.suffix.lower() == ".shp":
        parts = sorted(p.parent.glob(p.stem + ".*"))
        h = hashlib.sha256()
        for part in parts:
            h.update(part.name.encode())
            h.update(file_hash(part).encode())
        return h.hexdigest()
    return file_hash(p)

Hashing the sidecar names as well as their contents is what stops a missing .prj from going unnoticed.

Example 2 — prove two files hold the same data #

import geopandas as gpd

def same_data(a, b, decimals=9, rename_truncated=False):
    ga = gpd.read_parquet(a) if str(a).endswith(".parquet") else gpd.read_file(a)
    gb = gpd.read_parquet(b) if str(b).endswith(".parquet") else gpd.read_file(b)
    if rename_truncated and len(ga.columns) == len(gb.columns):
        gb.columns = list(ga.columns)          # shapefile truncates to 10 characters
    return content_hash(ga, decimals) == content_hash(gb, decimals)

The rename_truncated flag is not a hack to hide a difference — it is the explicit acknowledgement that a shapefile cannot carry a column called population_estimate_2019, which it stores as population.

Example 3 — a raster content hash #

import rasterio, hashlib, numpy as np

def raster_content_hash(path, round_to=None):
    with rasterio.open(path) as src:
        h = hashlib.sha256()
        h.update(str(src.crs).encode())
        h.update(np.array(src.transform[:6], dtype="float64").round(9).tobytes())
        h.update(f"{src.width}x{src.height}x{src.count}".encode())
        for i in range(1, src.count + 1):
            band = src.read(i)
            if round_to is not None and np.issubdtype(band.dtype, np.floating):
                band = np.round(band, round_to)
            nodata = src.nodatavals[i - 1]
            if nodata is not None and np.isnan(nodata):
                band = np.where(np.isnan(band), np.float64(-1e308), band)
            h.update(np.ascontiguousarray(band).tobytes())
        return h.hexdigest()

The NaN substitution matters: nan != nan, so two byte-identical arrays of NaN hash the same but two arrays that should be equal after a float operation may not. Replacing NaN with a sentinel before hashing makes the comparison say what you mean.

Explanation #

Why GeoPackage and GeoJSON are not byte-reproducible #

A GeoPackage records its own creation and last-change timestamps in gpkg_contents, so two writes a second apart differ. GeoJSON and FlatGeobuf as written by GDAL carry generator or timestamp information of their own. Shapefile and Parquet, in this test, produced identical bytes for identical input — which makes them the two formats where a byte hash is a usable output check.

Why the format's normalisation is the interesting part #

Every difference in the table above is a format doing something reasonable: declaring one geometry type per layer, ordering features spatially for fast reads, writing coordinates as decimal text. None is a bug, and all of them change the bytes. Canonicalising before hashing is how you compare the data through those choices.

Why coordinate precision has to be a decision #

Doubles have about 15–17 significant decimal digits, and a text round trip loses the last one or two. At UK latitudes a difference of 10⁻¹³ degrees is about 10 nanometres — physically meaningless and digest-changing. Choosing a precision is choosing how much agreement you require, and nine decimal places is about 0.1 mm, which is beyond any survey.

Why snapping before writing is the better fix #

Rounding at hash time hides the difference; snapping at write time removes it. A dataset whose coordinates have been snapped to a stated grid round-trips exactly through every format, hashes identically everywhere, and is honest about its own precision. The cost is one line in the writer.

Edge cases or notes #

• Hash the schema too if a column rename matters to you.
• Column order is storage. Sort columns before hashing, or the digest changes when someone reorders them.
• Empty and null geometries differ. Decide which you treat as equal.
• Z and M values are dropped by force_2d; keep them if they are data.
• Categorical dtypes stringify differently. Casting the whole table to str avoids surprises.
• Large datasets need a streaming hash. Hash per chunk and combine, in a fixed chunk order.
• SHA-256 is right here. MD5 is for detecting corruption, not for claims.
• Record the recipe with the digest. A hash without its parameters is not reproducible.

Internal links #

• Provenance and lineage explained for spatial workflows — where the digest is recorded
• Dataset versioning explained — checksums as version identity
• How to write a metadata record for a dataset in Python — the field the digest goes in
• Metadata disappears when you convert the file — the other side of format differences
• You cannot tell which data version produced a map — what the digest answers
• How to validate pipeline inputs and outputs in Python — hashing as a pipeline check
• Coordinate precision explained — choosing the rounding
• GeoParquet and columnar storage explained — one of the two reproducible formats

FAQ #

Why does hashing a GeoPackage give a different result each time? #

Because the container records its own timestamps. Two writes of the same layer a second apart produced different SHA-256 digests in a direct test; shapefile and Parquet were identical.

How do I prove two spatial files hold the same data? #

Hash the canonicalised content: normalise geometry type and ring order, sort the rows and columns, round coordinates to a stated precision, then hash the result.

Which formats are byte-reproducible? #

In testing, shapefile and Parquet were; GeoPackage, GeoJSON and FlatGeobuf were not.

What coordinate precision should I hash at? #

Nine decimal degrees is about 0.1 mm and safely below any survey precision. State whatever you choose alongside the digest.

Why did my GeoJSON hash differ from the source? #

Text formats do not round-trip doubles exactly — a round trip left only 4 of 120 geometries bit-identical, with a maximum displacement of about 10 nanometres. Snap the coordinates before writing, or round when hashing.

Should I use MD5 or SHA-256? #

SHA-256. MD5 is adequate for detecting accidental corruption and inadequate for anything you want to assert.