Metadata disappears when you convert the file

Problem statement

The GeoPackage had a title, a licence and a lineage note in its metadata. It was converted to a shapefile for a client, and now it has none of that — plus population_estimate_2019 is called population, population_rank is populati_1, and population_as_of is text rather than a date. The FlatGeobuf copy made for the web viewer kept the names, but every Polygon in it is now a MultiPolygon and the countries are in a different order.

No step reported an error. Conversions are lossy in ways that are specific to each driver, and the losses are silent because the driver is doing what the format requires. Measured on Natural Earth's 258 countries with eight typed attributes, the same GeoDataFrame written to five formats and read back (Example 1):

The shapefile turned country_name_long into country_na, population_estimate_2019 into population and gdp_million_usd into gdp_millio.

Quick answer

Read both sides with the same reader and diff them:

import geopandas as gpd
from osgeo import gdal
gdal.UseExceptions()

def conversion_report(src, dst):
    a, b = gpd.read_file(src), gpd.read_file(dst)
    da = gdal.OpenEx(str(src)).GetMetadata()
    db = gdal.OpenEx(str(dst)).GetMetadata()
    return {
        "dataset_metadata_lost": {k: v for k, v in da.items() if k not in db},
        "columns_renamed": {x: y for x, y in zip(a.columns, b.columns) if x != y},
        "dtype_changes": {c: (str(a[c].dtype), str(b[c].dtype))
                          for c in a.columns if c in b.columns and str(a[c].dtype) != str(b[c].dtype)},
        "geom_types": (sorted(a.geom_type.unique()), sorted(b.geom_type.unique())),
        "crs_same": a.crs == b.crs,
        "features": (len(a), len(b)),
    }

gpd.read_file("master.gpkg").to_file("client.shp")
for key, value in conversion_report("master.gpkg", "client.shp").items():
    print(f"{key:22} {value}")

dataset_metadata_lost  {'LICENCE': 'Public domain (Natural Earth)', 'LINEAGE': 'ne_10m_admin_0_countries; eight attributes renamed and typed', 'TITLE': 'Countries with population and GDP, 1:10m'}
columns_renamed        {'country_name_long': 'country_na', 'population_estimate_2019': 'population', 'population_rank': 'populati_1', 'gdp_million_usd': 'gdp_millio', 'income_group': 'income_gro', 'population_as_of': 'populati_2', 'is_sovereign': 'is_soverei'}
dtype_changes          {}
geom_types             (['MultiPolygon'], ['MultiPolygon', 'Polygon'])
crs_same               True
features               (258, 258)

dtype_changes is empty for a bad reason: it compares columns present on both sides, and every column changed its name. The date that became a string is in there, under populati_2. A report that matches columns by name needs the rename map first.

Run it once on your own pipeline. The surprise is rarely the format you were worried about.

Identify the loss, avoid shapefiles, copy metadata across

1. Identify which loss you have

• Column names truncated — shapefile, ten characters, no warning.
• Types changed — dates to strings in shapefile; 64-bit integers narrowed to 32-bit by GeoJSON's reader when the values fit.
• Geometry type promoted — GeoPackage and FlatGeobuf declare one layer type and promote Polygon to MultiPolygon.
• Feature order changed — FlatGeobuf sorts features by a packed Hilbert R-tree.
• Dataset metadata dropped — every conversion, unless you copy it.

2. Do not convert to shapefile if you can avoid it

Ten-character field names, no date type, a 2 GB limit per file and a separate .dbf encoding are not problems that can be worked around; they are the format. Where a client insists, agree a field-name mapping in advance and ship it with the data.

3. Copy dataset metadata explicitly

No driver carries a dataset-level record across a conversion. Read it before, write it after.

src, dst = "master.gpkg", "copy.gpkg"
before = gdal.OpenEx(src).GetMetadata()
gpd.read_file(src).to_file(dst)                     # the conversion
print("after conversion:", gdal.OpenEx(dst).GetMetadata())
ds = gdal.OpenEx(dst, gdal.OF_UPDATE | gdal.OF_VECTOR)
for k, v in before.items():
    ds.SetMetadataItem(k, v)
ds = None
print("after copying:   ", sorted(gdal.OpenEx(dst).GetMetadata()))

after conversion: {}
after copying:    ['LICENCE', 'LINEAGE', 'TITLE']

4. Pin the schema rather than hoping

If the target format cannot hold your column names, choose the truncations yourself so they are meaningful and stable, and publish the mapping.

RENAME = {"country_name_long": "cname", "population_estimate_2019": "pop2019",
          "population_rank": "pop_rank", "gdp_million_usd": "gdp_musd",
          "income_group": "income_grp", "population_as_of": "pop_asof", "is_sovereign": "sovereign"}
gdf = gpd.read_file("master.gpkg")
gdf.rename(columns=RENAME).to_file("out.shp")
print(list(gpd.read_file("out.shp").columns))

['cname', 'iso_a3', 'pop2019', 'pop_rank', 'gdp_musd', 'income_grp', 'pop_asof', 'sovereign', 'geometry']

pop2019 is a better name than population, and pop_rank is a better one than populati_1. They are ones you chose.

5. Decide what to do about dates

A shapefile stores dates as a DBF date field, and reading it back with GeoPandas gave a string column in the measurement above. If dates matter, write them as ISO text in a named column deliberately, so the shape of the loss is your decision rather than the driver's.

6. Normalise geometry types before writing

If the output format will promote everything to multi-part, do it yourself first. Then the file matches what you expect, and a checksum taken before and after agrees.

7. Check the conversion, every time, in the pipeline

The diff function above is twenty lines and catches all of the above. Make it a gate on the export step.

Measuring what a conversion loses

The data used on this page

The layer is Natural Earth's 1:10m admin-0 countries — 258 features, 151 MultiPolygons and 107 Polygons — with eight of its attributes renamed to the long, typed names an analysis would give them: the long country name, the ISO code, the 2019 population estimate (64-bit), the population rank (32-bit), GDP in millions of dollars, the income group, the year of the population estimate as a date, and whether the feature is a sovereign country. The frame is kept in memory as layer, and saved as master.gpkg with a title, licence and lineage in the GeoPackage's dataset metadata.

import io
import os
import zipfile

import geopandas as gpd
import pandas as pd
import requests
from osgeo import gdal

gdal.UseExceptions()
if not os.path.exists("ne_countries.shp"):
    archive = requests.get("https://naciscdn.org/naturalearth/10m/cultural/ne_10m_admin_0_countries.zip", timeout=120)
    with zipfile.ZipFile(io.BytesIO(archive.content)) as z:
        for ext in (".shp", ".shx", ".dbf", ".prj", ".cpg"):
            open(f"ne_countries{ext}", "wb").write(z.read(f"ne_10m_admin_0_countries{ext}"))
c = gpd.read_file("ne_countries.shp")
layer = gpd.GeoDataFrame({
        "country_name_long": c["NAME_LONG"], "iso_a3": c["ADM0_A3"],
        "population_estimate_2019": c["POP_EST"].astype("int64"),
        "population_rank": c["POP_RANK"].astype("int32"),
        "gdp_million_usd": c["GDP_MD"].astype("int64"), "income_group": c["INCOME_GRP"],
        "population_as_of": pd.to_datetime(c["POP_YEAR"].astype(str), format="%Y"),
        "is_sovereign": c["TYPE"].eq("Sovereign country"),
    }, geometry=c.geometry, crs=c.crs)
if not os.path.exists("master.gpkg"):
    layer.to_file("master.gpkg")
    ds = gdal.OpenEx("master.gpkg", gdal.OF_UPDATE | gdal.OF_VECTOR)
    ds.SetMetadataItem("TITLE", "Countries with population and GDP, 1:10m")
    ds.SetMetadataItem("LICENCE", "Public domain (Natural Earth)")
    ds.SetMetadataItem("LINEAGE", "ne_10m_admin_0_countries; eight attributes renamed and typed")
    ds = None
print(len(layer), "features;", layer.geom_type.value_counts().to_dict())
print(layer.dtypes.astype(str).to_dict())
print("master.gpkg:", gpd.read_file("master.gpkg").geom_type.value_counts().to_dict())
print(gdal.OpenEx("master.gpkg").GetMetadata())

258 features; {'MultiPolygon': 151, 'Polygon': 107}
{'country_name_long': 'str', 'iso_a3': 'str', 'population_estimate_2019': 'int64', 'population_rank': 'int32', 'gdp_million_usd': 'int64', 'income_group': 'str', 'population_as_of': 'datetime64[us]', 'is_sovereign': 'bool', 'geometry': 'geometry'}
master.gpkg: {'MultiPolygon': 258}
{'LICENCE': 'Public domain (Natural Earth)', 'LINEAGE': 'ne_10m_admin_0_countries; eight attributes renamed and typed', 'TITLE': 'Countries with population and GDP, 1:10m'}

Example 1 — measure the loss yourself

import geopandas as gpd, pandas as pd, pathlib
import pyarrow.parquet as pq
from osgeo import gdal

def round_trip_report(gdf, src_metadata, out_dir="rt"):
    out = pathlib.Path(out_dir); out.mkdir(exist_ok=True)
    cases = [("ESRI Shapefile", "a.shp"), ("GPKG", "b.gpkg"), ("GeoJSON", "c.geojson"),
             ("FlatGeobuf", "d.fgb"), (None, "e.parquet")]
    rows = []
    for driver, name in cases:
        p = out / name
        for f in p.parent.glob(p.stem + ".*"):
            f.unlink()
        gdf.to_parquet(p) if driver is None else gdf.to_file(p, driver=driver)
        back = gpd.read_parquet(p) if driver is None else gpd.read_file(p)
        pop = back.iloc[:, 2]                              # population_estimate_2019, whatever it is called now
        date = back.iloc[:, 6]                             # population_as_of
        if driver is None:                                 # GDAL's Parquet route may need libduckdb
            meta = {k.decode(): v for k, v in (pq.read_schema(p).metadata or {}).items()}
        else:
            meta = gdal.OpenEx(str(p)).GetMetadata() or {}
        rows.append({
            "format": p.suffix.lstrip("."),
            "names_intact": all(c in back.columns for c in gdf.columns),
            "longest_name": max(len(c) for c in back.columns),
            "int64_column": str(pop.dtype),
            "date_column": str(date.dtype),
            "geom_types": ",".join(sorted(back.geom_type.unique())),
            "order_kept": bool((back.iloc[:, 1].values == gdf.iloc[:, 1].values).all()),
            "metadata_kept": all(k in meta for k in src_metadata),
            "MB": round(sum(f.stat().st_size for f in p.parent.glob(p.stem + ".*")) / 1e6, 2),
        })
    return pd.DataFrame(rows)

report = round_trip_report(layer, gdal.OpenEx("master.gpkg").GetMetadata())
print(report.to_string(index=False))

 format  names_intact  longest_name int64_column    date_column           geom_types  order_kept  metadata_kept    MB
    shp         False            10        int64            str MultiPolygon,Polygon        True          False  8.89
   gpkg          True            24        int64 datetime64[ms]         MultiPolygon        True          False  9.14
geojson          True            24        int32 datetime64[ms] MultiPolygon,Polygon        True          False 24.48
    fgb          True            24        int64 datetime64[ms]         MultiPolygon       False          False  8.93
parquet          True            24        int64 datetime64[us] MultiPolygon,Polygon        True          False  6.74

Not one format carried the dataset metadata — not even GeoPackage to GeoPackage, because the metadata belongs to the file and GeoPandas moves only the layer.

The setup output shows the promotion had already happened once before any of this: master.gpkg holds 258 MultiPolygons, because writing the mixed layer to a GeoPackage was itself a conversion. And the shapefile in the Quick answer shows it running backwards — a shapefile does not distinguish a single-part MultiPolygon from a Polygon, so the single-part MultiPolygons came back as Polygons.

Example 2 — a lossless-enough export with the mapping shipped

import json, pathlib, geopandas as gpd

def export_shapefile(gdf, path, rename=None):
    rename = rename or {}
    auto = {c: c[:10] for c in gdf.columns if len(c) > 10 and c not in rename}
    if auto:
        raise ValueError(f"columns need explicit short names: {sorted(auto)}")
    out = gdf.rename(columns=rename)
    out.to_file(path, driver="ESRI Shapefile")
    pathlib.Path(str(path).replace(".shp", "_fields.json")).write_text(
        json.dumps({v: k for k, v in rename.items()}, indent=2))
    return path

gdf = gpd.read_file("master.gpkg")
try:
    export_shapefile(gdf, "client_strict.shp")
except ValueError as exc:
    print("refused:", exc)
export_shapefile(gdf, "client_strict.shp", rename=RENAME)
print(pathlib.Path("client_strict_fields.json").read_text()[:120], "…")

refused: columns need explicit short names: ['country_name_long', 'gdp_million_usd', 'income_group', 'is_sovereign', 'population_as_of', 'population_estimate_2019', 'population_rank']
{
  "cname": "country_name_long",
  "pop2019": "population_estimate_2019",
  "pop_rank": "population_rank",
  "gdp_musd" …

Raising rather than truncating is the point: it forces the mapping to be a decision, and the sidecar lets the client restore the original names.

Example 3 — a conversion gate

def assert_conversion_ok(src, dst, allow_rename=(), allow_promotion=True):
    r = conversion_report(src, dst)
    problems = []
    if r["features"][0] != r["features"][1]:
        problems.append(f"feature count changed: {r['features']}")
    if not r["crs_same"]:
        problems.append("CRS changed")
    unexpected = {k: v for k, v in r["columns_renamed"].items() if k not in allow_rename}
    if unexpected:
        problems.append(f"unexpected renames: {unexpected}")
    if r["dataset_metadata_lost"]:
        problems.append(f"dataset metadata lost: {sorted(r['dataset_metadata_lost'])}")
    if not allow_promotion and r["geom_types"][0] != r["geom_types"][1]:
        problems.append(f"geometry types changed: {r['geom_types']}")
    if problems:
        raise ValueError(f"{src} → {dst}: " + "; ".join(problems))
    return True

for dst in ("client.shp", "copy.gpkg", "rt/d.fgb"):
    try:
        assert_conversion_ok("master.gpkg", dst, allow_promotion=False)
        print(dst, "passed")
    except ValueError as exc:
        print("FAILED", exc)

FAILED master.gpkg → client.shp: unexpected renames: {'country_name_long': 'country_na', 'population_estimate_2019': 'population', 'population_rank': 'populati_1', 'gdp_million_usd': 'gdp_millio', 'income_group': 'income_gro', 'population_as_of': 'populati_2', 'is_sovereign': 'is_soverei'}; dataset metadata lost: ['LICENCE', 'LINEAGE', 'TITLE']; geometry types changed: (['MultiPolygon'], ['MultiPolygon', 'Polygon'])
copy.gpkg passed
FAILED master.gpkg → rt/d.fgb: dataset metadata lost: ['LICENCE', 'LINEAGE', 'TITLE']

copy.gpkg passes because Step 3 copied its metadata across; the other two fail on exactly the losses the table measured. The FlatGeobuf failed only on metadata — from an all-MultiPolygon master there was nothing left to promote.

Drivers do not carry metadata across

Why drivers do not carry metadata across

A conversion is a read and a write through two independent drivers. The reader hands GDAL a layer definition and features; the writer asks what the target format can store. Dataset-level metadata is not part of that handoff unless the tool doing the conversion explicitly copies it, and most do not — ogr2ogr does for some driver pairs and not for others.

Why geometry promotion happens

GeoPackage and FlatGeobuf declare a geometry type per layer. A layer holding both Polygon and MultiPolygon must be declared as MultiPolygon, and every Polygon is wrapped on write. In the test data — 151 MultiPolygons and 107 Polygons — both formats returned 258 MultiPolygons. Shapefile, GeoJSON and Parquet preserved the mix.

Why FlatGeobuf reorders features

The format stores a packed Hilbert R-tree so that a reader can do a spatial query without an index file. Building it requires sorting the features spatially, so the file's order is not the write order. That is a feature, and it breaks any code that relies on row position.

Why GeoJSON is the largest and the least typed

It is text: a coordinate that is eight bytes as a double is up to eighteen characters as a decimal, which is why the GeoJSON in the table is 3.6 times the size of the Parquet. It also has only one number type, so integer widths are inferred by the reader rather than declared by the file — here the 2019 population estimates, which all fit in 32 bits, came back as int32.

Truncation collisions, encodings and field widths

• Two columns can truncate to the same name. GDAL disambiguates by numbering: population_estimate_2019, population_rank and population_as_of became population, populati_1 and populati_2, so which is which depends on the column order.
• .cpg carries the shapefile encoding. Without it, non-ASCII attributes are a lottery.
• Field widths are fixed in DBF. Long strings are silently cut.
• Booleans survived all five formats in this run, with GDAL 3.12 and pyogrio; a DBF stores them as a one-character logical field, so check what your own reader returns.
• Nulls in integer columns force floats unless the format has a nullable integer.
• Layer names change. A shapefile's layer name is its filename.
• Z values may be dropped. Check has_z before and after.
• Convert once, from the master. Chained conversions compound every loss.

Internal links

• How to read the metadata already inside your spatial files — the reader both sides of the diff use
• GIS file formats compared — choosing a target that loses less
• Shapefile column names are truncated — the most common single loss
• How to store metadata inside a GeoPackage — restoring the record after conversion
• How to checksum spatial datasets so you can prove they match — comparing content through a conversion
• Shapefile encoding errors in Python — the .cpg problem
• Mixed geometry types cause a save error — the other half of geometry promotion
• How to batch convert shapefiles to GeoPackage — converting in the useful direction

FAQ

Why did my column names change when I saved a shapefile?

The DBF format limits field names to ten characters. population_estimate_2019 became population and country_name_long became country_na, with no warning.

Which formats keep dataset-level metadata?

GeoPackage, GeoTIFF, FlatGeobuf and Parquet have somewhere to put it; GeoJSON and shapefile do not. But having somewhere is not the same as copying: in the measurement here, none of the five formats received the GeoPackage's title, licence or lineage when the layer was written through GeoPandas — not even another GeoPackage.

Why is my GeoJSON so much bigger?

Because it is text. The same 258-feature layer was 24.48 MB as GeoJSON and 6.74 MB as Parquet.

Why did the feature order change after writing FlatGeobuf?

The format sorts features into a packed Hilbert R-tree so readers can query spatially without an index. Row position is not preserved.

How do I stop silent losses?

Diff both sides of every conversion in the pipeline — feature count, CRS, column names, dtypes, geometry types and dataset metadata — and fail on anything you did not explicitly allow.