⚙️ How-To Metadata, licensing & provenance Files & formats Metadata, Licensing & Provenance · step 7 of 20 Updated

How to write a metadata record for a dataset in Python

Problem statement

Writing metadata by hand fails for a reason that has nothing to do with discipline: most of the fields are facts about the data, so a human copying them in is a human introducing errors. Extents drift after a clip, feature counts go stale after a filter, and the CRS in the record is the one the dataset had two versions ago.

The fix is to split the record in two. The derivable half — extent, CRS, schema, counts, checksums, geometry types — is computed at write time from the file being written. The authored half — title, what one feature is, lineage, licence, contact — is a small YAML file a person maintains, and a gate refuses to publish without it.

This guide builds both halves and the gate.

Quick answer 

import geopandas as gpd, hashlib, pathlib, datetime, json, yaml

def write_metadata(data_path, authored_path, out_path=None, layer=None):
    gdf = gpd.read_file(data_path, layer=layer)
    authored = yaml.safe_load(pathlib.Path(authored_path).read_text())

    REQUIRED = ["title", "feature_definition", "content_date", "lineage", "licence", "contact"]
    missing = [k for k in REQUIRED if not authored.get(k)]
    if missing:
        raise ValueError(f"{authored_path}: missing {missing}")

    b = gdf.total_bounds
    record = {
        **authored,
        "derived": {
            "file": pathlib.Path(data_path).name,
            "layer": layer,
            "features": int(len(gdf)),
            "geometry_types": sorted(gdf.geom_type.dropna().unique().tolist()),
            "crs": gdf.crs.to_string() if gdf.crs is not None else None,
            "bbox": [round(float(v), 6) for v in b],
            "schema": {c: str(t) for c, t in gdf.dtypes.items() if c != gdf.geometry.name},
            "null_counts": {c: int(gdf[c].isna().sum())
                            for c in gdf.columns if c != gdf.geometry.name},
            "generated": datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds"),
        },
    }
    out = pathlib.Path(out_path or (str(data_path) + ".meta.json"))
    out.write_text(json.dumps(record, indent=2, default=str))
    return record

The derived block is regenerated on every write, so it cannot disagree with the data. The authored block is reviewed in a pull request like code.

Two panels separating the derived half of a metadata record from the authored half.
Anything a program can compute should never be typed; anything it cannot must be gated.

Step-by-step solution

1. Write the authored half first, as YAML next to the data 

title: Coastal flood risk zones, East Sussex
feature_definition: >
  One polygon per contiguous modelled inundation area at the 1-in-200-year
  return period. Areas under 100 m² removed.
content_date: 2025-11-01
lineage:
  - Environment Agency LiDAR Composite DTM 2024, 2 m
  - pysheds 0.4 breach_depressions → flowdir(D8) → accumulation
  - Return period surface from EA fluvial model v7, 200-year
licence: OGL-UK-3.0
attribution: Contains Environment Agency data © Crown copyright and database right 2025
contact: [email protected]
accuracy:
  positional_rmse_m: 2.0
  method: 43 GNSS control points, June 2025

YAML rather than JSON because a human edits it, and multi-line strings are readable.

2. Derive the rest at write time

Compute the derived block from the file you just wrote, not from the in-memory frame — that is what catches a writer that dropped a column or promoted a geometry type.

3. Include the schema and the null counts

The schema is what breaks consumers when it changes, and the null counts tell a user whether a column is worth reading. Both are one line to compute and are the fields people thank you for.

4. Add a content checksum, not a byte checksum

A byte hash of a GeoPackage changes every time you write it. Hash the canonicalised data instead — see How to checksum spatial datasets so you can prove they match.

5. Embed the record where the format allows

A sidecar is easy to separate from the data. Put at least the title, licence, version and content date inside the file as well, so a copy on somebody's laptop still knows what it is. How to store metadata inside a GeoPackage and How to embed provenance tags in a GeoTIFF cover the two formats that make this easy.

6. Emit the standard shapes from the same record

Do not author STAC or ISO directly; generate them. Metadata standards compared has the mapping.

7. Gate the publish step

The check is three lines and it is the only thing that keeps records complete once the deadline is close.

Flow from authored YAML and written data through derivation and validation to a sidecar, embedded tags and a STAC item.
One authored file in, three artefacts out, none of them typed twice.

Code examples

Example 1 — derive a raster's block too 

import rasterio, numpy as np

def derived_raster(path):
    with rasterio.open(path) as src:
        stats = []
        for i in range(1, src.count + 1):
            band = src.read(i, masked=True)
            stats.append({
                "band": i,
                "dtype": src.dtypes[i - 1],
                "nodata": src.nodatavals[i - 1],
                "valid_fraction": float(band.count() / band.size),
                "min": None if band.count() == 0 else float(band.min()),
                "max": None if band.count() == 0 else float(band.max()),
                "mean": None if band.count() == 0 else float(band.mean()),
            })
        return {
            "size": [src.width, src.height],
            "bands": src.count,
            "crs": src.crs.to_string() if src.crs else None,
            "transform": list(src.transform)[:6],
            "pixel_size": [abs(src.transform.a), abs(src.transform.e)],
            "bbox": list(src.bounds),
            "band_stats": stats,
            "tags": src.tags(),
        }

valid_fraction is the raster equivalent of a null count and answers "is this tile mostly nodata?" without opening it in a viewer.

Example 2 — a schema block that can be diffed between versions 

def schema_block(gdf):
    out = {}
    for col, dtype in gdf.dtypes.items():
        if col == gdf.geometry.name:
            continue
        s = gdf[col]
        entry = {"type": str(dtype), "nulls": int(s.isna().sum())}
        if s.dtype == object or str(s.dtype) == "string":
            uniques = s.dropna().unique()
            if len(uniques) <= 25:
                entry["values"] = sorted(map(str, uniques))
            entry["max_length"] = int(s.dropna().astype(str).str.len().max() or 0)
        elif "int" in str(s.dtype) or "float" in str(s.dtype):
            entry["min"], entry["max"] = float(s.min()), float(s.max())
        out[col] = entry
    return out

Recording the value list for low-cardinality columns turns a code list into something a consumer can validate against, and makes a new category visible in a diff.

Example 3 — the publish gate 

REQUIRED = ["title", "feature_definition", "content_date", "lineage", "licence", "contact"]
KNOWN_LICENCES = {"CC0-1.0", "CC-BY-4.0", "CC-BY-SA-4.0", "ODbL-1.0",
                  "ODC-BY-1.0", "OGL-UK-3.0", "proprietary"}

def gate(record):
    problems = []
    problems += [f"missing {k}" for k in REQUIRED if not record.get(k)]
    if record.get("licence") not in KNOWN_LICENCES:
        problems.append(f"licence {record.get('licence')!r} is not a known identifier")
    d = record.get("derived", {})
    if not d.get("crs"):
        problems.append("dataset has no CRS")
    if d.get("features", 0) == 0:
        problems.append("dataset is empty")
    bbox = d.get("bbox") or [0, 0, 0, 0]
    if bbox[0] > bbox[2] or bbox[1] > bbox[3]:
        problems.append(f"bbox is not in west, south, east, north order: {bbox}")
    if problems:
        raise ValueError("metadata gate failed:\n  " + "\n  ".join(problems))
    return True

The bbox order check is worth having explicitly: a STAC validator will accept a reversed bounding box without complaint.

Explanation

Why the derived block must come from the written file

The file is what ships, and writers change things: a shapefile truncates column names to ten characters, a GeoPackage promotes every Polygon to MultiPolygon, a GeoJSON round trip perturbs coordinates in the thirteenth decimal place. A record derived from the in-memory frame documents a dataset that nobody received.

Why the authored fields need a gate rather than a template

A template with empty fields ships with empty fields. The only mechanism that reliably produces complete records is one that stops the publish, because that is the only moment at which somebody has both the knowledge and the motivation to fill them in.

Why null counts and value lists earn their place

They are the two things a consumer would otherwise compute themselves, and they are the two that change silently between versions. A column that was 2% null and is now 60% null is a broken upstream join, and the metadata diff is where it shows up first.

Why YAML for the authored half

The authored half is a small document with multi-line prose in it, maintained by a person and reviewed in a pull request. JSON is unpleasant to write multi-line strings in; YAML is not. The generated record is JSON because programs read it.

Checklist of metadata gate checks: required fields, a known licence identifier, a CRS, a non-empty dataset, bbox ordering, and why warning instead of raising fails.
The bbox order check is here because no schema validator performs it.

Edge cases or notes

  • Multi-layer files need a record per layer. Plus one for the container.
  • total_bounds on an empty frame returns NaNs. Guard it.
  • Round the bbox deliberately. Six decimal places is ample and keeps the record diffable.
  • Datetime objects are not JSON. Pass default=str or convert explicitly.
  • Do not store credentials in lineage. Record the endpoint, not the token.
  • Keep the authored file in version control even when the data is not.
  • Regenerate on every release. A record that is older than the data is worse than none.
  • A record is a deliverable. Ship it in the zip.

FAQ

How do I generate metadata for a spatial dataset in Python?

Compute the derivable fields — extent, CRS, schema, counts, geometry types, checksum — from the written file, and merge them with a small authored YAML file holding title, feature definition, lineage, licence and contact.

Which fields should a person write?

Title, what one feature represents, the content date, lineage, licence, attribution and a contact. Everything else can be derived.

Should the record be JSON or YAML?

Author in YAML because a human maintains it; generate JSON because programs consume it.

Why derive from the file rather than the DataFrame?

Because writers change things — shapefiles truncate column names, GeoPackages promote geometry types — and the record must describe what was actually shipped.

How do I stop records from going stale?

Regenerate the derived block on every write, and gate the publish step so an incomplete authored block raises rather than warns.

What checksum should go in the record?

A content hash of the canonicalised data. A byte hash of a GeoPackage or GeoJSON changes between identical writes.