Metadata disappears when you convert the file #

Problem statement #

The GeoPackage had a title, a licence and a lineage note in its metadata table. It was converted to a shapefile for a client, and now it has none of that — plus population_estimate_2019 is called population, surveyed_on is text rather than a date, and every Polygon that used to be a Polygon is now a MultiPolygon.

No step reported an error. Conversions are lossy in ways that are specific to each driver, and the losses are silent because the driver is doing what the format requires. Measured on a 258-feature test layer with seven attributes, the same data written to five formats:

The shapefile turned country_name_long into country_na, population_estimate_2019 into population and fraction_urban into fraction_u.

Quick answer #

Read both sides with the same reader and diff them:

import geopandas as gpd
from osgeo import gdal
gdal.UseExceptions()

def conversion_report(src, dst):
    a, b = gpd.read_file(src), gpd.read_file(dst)
    da = gdal.OpenEx(str(src)).GetMetadata()
    db = gdal.OpenEx(str(dst)).GetMetadata()
    return {
        "dataset_metadata_lost": {k: v for k, v in da.items() if k not in db},
        "columns_renamed": {x: y for x, y in zip(a.columns, b.columns) if x != y},
        "dtype_changes": {c: (str(a[c].dtype), str(b[c].dtype))
                          for c in a.columns if c in b.columns and str(a[c].dtype) != str(b[c].dtype)},
        "geom_types": (sorted(a.geom_type.unique()), sorted(b.geom_type.unique())),
        "crs_same": a.crs == b.crs,
        "features": (len(a), len(b)),
    }

Run it once on your own pipeline. The surprise is rarely the format you were worried about.

Step-by-step solution #

1. Identify which loss you have #

• Column names truncated — shapefile, ten characters, no warning.
• Types changed — dates to strings in shapefile; integer widths narrowed by GeoJSON's reader.
• Geometry type promoted — GeoPackage and FlatGeobuf declare one layer type and promote Polygon to MultiPolygon.
• Feature order changed — FlatGeobuf sorts features by a packed Hilbert R-tree.
• Dataset metadata dropped — every conversion, unless you copy it.

2. Do not convert to shapefile if you can avoid it #

Ten-character field names, no date type, a 2 GB limit per file and a separate .dbf encoding are not problems that can be worked around; they are the format. Where a client insists, agree a field-name mapping in advance and ship it with the data.

3. Copy dataset metadata explicitly #

No driver carries a dataset-level record across a conversion. Read it before, write it after.

before = gdal.OpenEx(src).GetMetadata()
# … convert …
ds = gdal.OpenEx(dst, gdal.OF_UPDATE | gdal.OF_VECTOR)
for k, v in before.items():
    ds.SetMetadataItem(k, v)
ds = None

4. Pin the schema rather than hoping #

If the target format cannot hold your column names, choose the truncations yourself so they are meaningful and stable, and publish the mapping.

RENAME = {"country_name_long": "cname", "population_estimate_2019": "pop2019",
          "fraction_urban": "urb_frac", "surveyed_on": "survey_dt"}
gdf.rename(columns=RENAME).to_file("out.shp")

pop2019 is a better name than population, and it is one you chose.

5. Decide what to do about dates #

A shapefile stores dates as a DBF date field, and GDAL reads that back as a string in several configurations. If dates matter, write them as ISO text in a named column deliberately, so the shape of the loss is your decision rather than the driver's.

6. Normalise geometry types before writing #

If the output format will promote everything to multi-part, do it yourself first. Then the file matches what you expect, and a checksum taken before and after agrees.

7. Check the conversion, every time, in the pipeline #

The diff function above is twenty lines and catches all of the above. Make it a gate on the export step.

Code examples #

Example 1 — measure the loss yourself #

import geopandas as gpd, pandas as pd, numpy as np, pathlib

def round_trip_report(gdf, out_dir="rt"):
    out = pathlib.Path(out_dir); out.mkdir(exist_ok=True)
    cases = [("ESRI Shapefile", "a.shp"), ("GPKG", "b.gpkg"), ("GeoJSON", "c.geojson"),
             ("FlatGeobuf", "d.fgb"), (None, "e.parquet")]
    rows = []
    for driver, name in cases:
        p = out / name
        for f in p.parent.glob(p.stem + ".*"):
            f.unlink()
        gdf.to_parquet(p) if driver is None else gdf.to_file(p, driver=driver)
        back = gpd.read_parquet(p) if driver is None else gpd.read_file(p)
        rows.append({
            "format": p.suffix.lstrip("."),
            "names_intact": all(c in back.columns for c in gdf.columns),
            "longest_name": max(len(c) for c in back.columns),
            "geom_types": ",".join(sorted(back.geom_type.unique())),
            "bytes": sum(f.stat().st_size for f in p.parent.glob(p.stem + ".*")),
        })
    return pd.DataFrame(rows)

Example 2 — a lossless-enough export with the mapping shipped #

import json, pathlib, geopandas as gpd

def export_shapefile(gdf, path, rename=None):
    rename = rename or {}
    auto = {c: c[:10] for c in gdf.columns if len(c) > 10 and c not in rename}
    if auto:
        raise ValueError(f"columns need explicit short names: {sorted(auto)}")
    out = gdf.rename(columns=rename)
    out.to_file(path, driver="ESRI Shapefile")
    pathlib.Path(str(path).replace(".shp", "_fields.json")).write_text(
        json.dumps({v: k for k, v in rename.items()}, indent=2))
    return path

Raising rather than truncating is the point: it forces the mapping to be a decision, and the sidecar lets the client restore the original names.

Example 3 — a conversion gate #

def assert_conversion_ok(src, dst, allow_rename=(), allow_promotion=True):
    r = conversion_report(src, dst)
    problems = []
    if r["features"][0] != r["features"][1]:
        problems.append(f"feature count changed: {r['features']}")
    if not r["crs_same"]:
        problems.append("CRS changed")
    unexpected = {k: v for k, v in r["columns_renamed"].items() if k not in allow_rename}
    if unexpected:
        problems.append(f"unexpected renames: {unexpected}")
    if r["dataset_metadata_lost"]:
        problems.append(f"dataset metadata lost: {sorted(r['dataset_metadata_lost'])}")
    if not allow_promotion and r["geom_types"][0] != r["geom_types"][1]:
        problems.append(f"geometry types changed: {r['geom_types']}")
    if problems:
        raise ValueError(f"{src} → {dst}: " + "; ".join(problems))
    return True

Explanation #

Why drivers do not carry metadata across #

A conversion is a read and a write through two independent drivers. The reader hands GDAL a layer definition and features; the writer asks what the target format can store. Dataset-level metadata is not part of that handoff unless the tool doing the conversion explicitly copies it, and most do not — ogr2ogr does for some driver pairs and not for others.

Why geometry promotion happens #

GeoPackage and FlatGeobuf declare a geometry type per layer. A layer holding both Polygon and MultiPolygon must be declared as MultiPolygon, and every Polygon is wrapped on write. In the test data — 64 MultiPolygons and 56 Polygons — both formats returned 120 MultiPolygons. Shapefile, GeoJSON and Parquet preserved the mix.

Why FlatGeobuf reorders features #

The format stores a packed Hilbert R-tree so that a reader can do a spatial query without an index file. Building it requires sorting the features spatially, so the file's order is not the write order. That is a feature, and it breaks any code that relies on row position.

Why GeoJSON is the largest and the least typed #

It is text: a coordinate that is eight bytes as a double is eighteen characters as a decimal, which is why the GeoJSON in the table is nearly four times the size of the Parquet. It also has only one number type, so integer widths are inferred by the reader rather than declared by the file.

Edge cases or notes #

• Two columns can truncate to the same name. GDAL disambiguates, unpredictably.
• .cpg carries the shapefile encoding. Without it, non-ASCII attributes are a lottery.
• Field widths are fixed in DBF. Long strings are silently cut.
• Booleans become integers in most formats.
• Nulls in integer columns force floats unless the format has a nullable integer.
• Layer names change. A shapefile's layer name is its filename.
• Z values may be dropped. Check has_z before and after.
• Convert once, from the master. Chained conversions compound every loss.

Internal links #

• How to read the metadata already inside your spatial files — the reader both sides of the diff use
• GIS file formats compared — choosing a target that loses less
• Shapefile column names are truncated — the most common single loss
• How to store metadata inside a GeoPackage — restoring the record after conversion
• How to checksum spatial datasets so you can prove they match — comparing content through a conversion
• Shapefile encoding errors in Python — the .cpg problem
• Mixed geometry types cause a save error — the other half of geometry promotion
• How to batch convert shapefiles to GeoPackage — converting in the useful direction

FAQ #

Why did my column names change when I saved a shapefile? #

The DBF format limits field names to ten characters. population_estimate_2019 became population and country_name_long became country_na, with no warning.

Which formats keep dataset-level metadata? #

GeoPackage, GeoTIFF, FlatGeobuf and Parquet have somewhere to put it. GeoJSON and shapefile do not, and no driver copies a record across a conversion for you.

Why are all my polygons MultiPolygons now? #

GeoPackage and FlatGeobuf declare one geometry type per layer, so a mixed layer is declared MultiPolygon and every Polygon is wrapped on write.

Why is my GeoJSON so much bigger? #

Because it is text. The same 258-feature layer was 24.5 MB as GeoJSON and 6.7 MB as Parquet.

Why did the feature order change after writing FlatGeobuf? #

The format sorts features into a packed Hilbert R-tree so readers can query spatially without an index. Row position is not preserved.

How do I stop silent losses? #

Diff both sides of every conversion in the pipeline — feature count, CRS, column names, dtypes, geometry types and dataset metadata — and fail on anything you did not explicitly allow.