How to Build an Inventory of a GIS Data Folder in Python
On this page
Problem statement
Someone hands you a folder: a delivery of GIS data with nested subfolders, a mixture of shapefiles, GeoPackages, GeoJSON, GML and rasters, and no documentation. Before you can process any of it you need to answer basic questions:
- how many layers are there, and how many features in total?
- what CRSs are in use — is it one, or eleven?
- which files are broken, empty, or missing their sidecars?
- which are duplicates of each other under different names?
- what geometry types, and what attribute schemas?
Opening them one by one in QGIS takes a day. An inventory script takes a minute to write and seconds to run, and it produces a table you can sort, filter, share and re-run when the next delivery lands.
The trick is doing it cheaply: reading every feature of every file to count them is slow and usually unnecessary. Metadata reads answer most questions without touching the geometry — for most formats. The example folder on this page is built from real downloads for Rutland: 18 OpenStreetMap shapefiles, a GeoPackage, the Land Registry's parcels as GML and GeoJSON, ward boundaries, and an elevation raster.
Quick answer
Walk the tree, read metadata only, and write a table:
- find candidate files by extension, skipping sidecars and temp files
- read metadata with
pyogrio.read_info()— feature count, CRS, geometry type, fields - catch per-file errors so one bad file cannot end the scan
- add file-level facts: size, modification time, checksum
- write a CSV or Parquet manifest, and summarise the CRSs and types
VECTOR_EXT = {".shp", ".gpkg", ".geojson", ".json", ".fgb", ".gml", ".kml", ".tab"}
rows = []
for path in sorted(root.rglob("*")):
if not path.is_file() or path.suffix.lower() not in VECTOR_EXT:
continue
try:
for name, _ in pyogrio.list_layers(path):
info = pyogrio.read_info(path, layer=name)
rows.append({"file": str(path.relative_to(root)), "layer": name, "features": info["features"],
"geometry": info["geometry_type"], "crs": info["crs"], "fields": len(info["fields"]),
"size_mb": round(path.stat().st_size / 1e6, 2)})
except Exception as exc:
rows.append({"file": str(path.relative_to(root)), "layer": None, "error": f"{type(exc).__name__}: {exc}"})
inv = pd.DataFrame(rows)
inv.to_csv("out/inventory_quick.csv", index=False)
print(len(inv), "layers;", inv.groupby("crs", dropna=False)["features"].agg(["count", "sum"]).to_string())26 layers; count sum
crs
EPSG:27700 2 42506
EPSG:4326 23 28664
NaN 1 300
The layer with no CRS is the deliberately broken copy; everything else is WGS 84 or British National Grid — two CRSs, which is worth knowing before anything gets joined.
What an inventory records
Discover, read metadata, flag problems, write it down
Discover the real data files
A folder tree contains far more than data: sidecars, lock files, cloud placeholders, and your own outputs.
VECTOR_EXT = {".shp", ".gpkg", ".geojson", ".json", ".fgb", ".gml", ".kml", ".tab", ".mif"}
RASTER_EXT = {".tif", ".tiff", ".img", ".asc", ".vrt", ".jp2", ".nc"}
SKIP_PARTS = {"__pycache__", ".git", ".ipynb_checkpoints", "$RECYCLE.BIN"}
def candidates(folder: Path):
for p in sorted(folder.rglob("*")):
if not p.is_file() or p.name.startswith((".", "~", "$")):
continue
if any(part in SKIP_PARTS for part in p.parts):
continue
ext = p.suffix.lower()
if ext in VECTOR_EXT:
yield p, "vector"
elif ext in RASTER_EXT:
yield p, "raster"
found = list(candidates(root))
all_files = [p for p in root.rglob("*") if p.is_file()]
print(f"{len(all_files)} files, {len(found)} candidates "
f"({sum(k == 'vector' for _, k in found)} vector, {sum(k == 'raster' for _, k in found)} raster)")
print("skipped by extension:", pd.Series([p.suffix for p in all_files if (p, 'vector') not in found
and (p, 'raster') not in found]).value_counts().to_dict())102 files, 25 candidates (24 vector, 1 raster)
skipped by extension: {'.cpg': 19, '.shx': 19, '.dbf': 19, '.prj': 18, '.gfs': 1, '': 1}
Most of the folder is sidecars: each shapefile brings a .shx, .dbf, .prj and .cpg, and reading the GML once left a .gfs beside it — GDAL's cached description of its schema. Note that a File Geodatabase (.gdb) is a directory, and a GeoPackage may hold many layers — so files and layers are not the same thing, which is why the inventory has a row per layer.
Read layer metadata without loading features
def vector_rows(path):
out = []
for layer_name, geom_type in pyogrio.list_layers(path):
start = time.perf_counter()
info = pyogrio.read_info(path, layer=layer_name)
out.append({
"layer": layer_name, "features": int(info["features"]),
"geometry_type": info["geometry_type"] or geom_type, "crs": info["crs"],
"field_count": len(info["fields"]), "fields": ",".join(info["fields"][:12]),
"bounds": [round(v, 4) for v in info["total_bounds"]] if info.get("total_bounds") else None,
"fast_count": info["capabilities"]["fast_feature_count"],
"read_ms": round((time.perf_counter() - start) * 1000, 1),
})
return out
timing = []
for path in [root / "atlas.gpkg", root / "osm/gis_osm_buildings_a_free_1.shp",
root / "parcels/Land_Registry_Cadastral_Parcels.gml", root / "parcels/rutland_parcels.geojson"]:
for row in vector_rows(path)[:1]:
timing.append({"file": path.name, "MB": round(path.stat().st_size / 1e6, 1), **{k: row[k] for k in
["features", "fast_count", "read_ms"]}})
print(pd.DataFrame(timing).to_string(index=False)) file MB features fast_count read_ms
atlas.gpkg 3.6 2405 True 4.2
gis_osm_buildings_a_free_1.shp 1.0 5513 True 0.4
Land_Registry_Cadastral_Parcels.gml 30.5 21253 True 0.3
rutland_parcels.geojson 43.2 21253 True 290.2
read_info() returns the feature count, CRS, geometry type, field names and dtypes, the layer extent, and — usefully — the driver's capabilities. For the GeoPackage and the shapefile the count comes from the header, and the read takes a millisecond whatever the file size. GeoJSON has no header: GDAL's GeoJSON driver parses the whole file when it opens it, so it reports a fast count, but only after spending 317 ms on 43 MB — hundreds of times longer, and growing with the file. The GML is fast only because GDAL wrote a .gfs file beside it on the first read, which records the count; the very first read parses the file too. For a folder of large GeoJSON or GML files, the inventory is no longer cheap.
Add raster metadata
def raster_row(path):
with rasterio.open(path) as src:
return {
"layer": path.stem, "geometry_type": "raster",
"crs": src.crs.to_string() if src.crs else None,
"width": src.width, "height": src.height, "bands": src.count, "dtype": src.dtypes[0],
"pixel_size": round(abs(src.transform.a), 6), "nodata": src.nodata,
"bounds": [round(v, 4) for v in src.bounds], "overviews": len(src.overviews(1)),
}
print(raster_row(root / "terrain/copernicus_dem_N52_W001.tif")){'layer': 'copernicus_dem_N52_W001', 'geometry_type': 'raster', 'crs': 'EPSG:4326', 'width': 2400, 'height': 3600, 'bands': 1, 'dtype': 'float32', 'pixel_size': 0.000417, 'nodata': None, 'bounds': [-1.0002, 52.0001, -0.0002, 53.0001], 'overviews': 3}
Reading a raster's profile is likewise a header operation — no pixels are decoded. The Copernicus tile is 2,400 by 3,600 cells — at this latitude the product stores longitude at 1.5 arc-seconds and latitude at 1 — with three overview levels already built, which is what a Cloud-Optimised GeoTIFF looks like. It declares no nodata value.
Record the file-level facts
import hashlib
def file_row(path):
stat = path.stat()
return {"path": str(path.relative_to(root)), "name": path.name, "ext": path.suffix.lower(),
"size_mb": round(stat.st_size / 1e6, 3),
"modified": datetime.fromtimestamp(stat.st_mtime, timezone.utc).isoformat(timespec="seconds")}
def quick_hash(path: Path, chunk=1 << 20) -> str:
"""Hash the size plus the first and last MB — fast, good enough to spot copies."""
size = path.stat().st_size
h = hashlib.sha1(str(size).encode())
with open(path, "rb") as fh:
h.update(fh.read(chunk))
if size > 2 * chunk:
fh.seek(-chunk, 2)
h.update(fh.read(chunk))
return h.hexdigest()[:16]
for name in ["boundaries/rutland_wards.geojson", "boundaries/rutland_wards_FINAL.geojson"]:
print(file_row(root / name)["name"], quick_hash(root / name))rutland_wards.geojson 6c2fd33cb7df80f5
rutland_wards_FINAL.geojson 6c2fd33cb7df80f5
A partial hash finds the copy-of-a-copy duplicates that plague delivery folders without reading every byte. It is a hint, not proof — two files could share a size, head and tail and differ in the middle — so confirm a match with a full hash before deleting anything.
Flag the problems you can detect cheaply
REQUIRED_SIDECARS = {".shp": (".shx", ".dbf")}
def quality_flags(path: Path, row: dict) -> list[str]:
flags = [f"missing {ext}" for ext in REQUIRED_SIDECARS.get(path.suffix.lower(), ())
if not path.with_suffix(ext).exists()]
if path.suffix.lower() == ".shp" and not path.with_suffix(".prj").exists():
flags.append("no .prj")
if row.get("kind") == "vector" and not row.get("crs"):
flags.append("no CRS")
if row.get("features") == 0:
flags.append("empty layer")
if row.get("bounds") and row["bounds"][0] == row["bounds"][2] and row["bounds"][1] == row["bounds"][3]:
flags.append("zero extent")
return flagsFlags turn the inventory from a description into a to-do list.
Write the manifest and summarise it
rows = []
for path, kind in candidates(root):
try:
layers = vector_rows(path) if kind == "vector" else [raster_row(path)]
rows += [{**file_row(path), "kind": kind, "hash": quick_hash(path), **layer, "error": ""} for layer in layers]
except Exception as exc:
rows.append({**file_row(path), "kind": kind, "error": f"{type(exc).__name__}: {exc}"})
for row in rows:
row["flags"] = "; ".join(quality_flags(root / row["path"], row))
inv = pd.DataFrame(rows)
inv.to_csv("out/inventory.csv", index=False)
inv.to_parquet("out/inventory.parquet")
print(inv.groupby("crs", dropna=False).agg(layers=("layer", "size"), features=("features", "sum")).to_string())
print(inv["geometry_type"].value_counts(dropna=False).to_dict())
print(inv.loc[inv["flags"] != "", ["path", "flags"]].to_string(index=False))
print(f"total: {len(inv)} layers, {inv['features'].sum():,.0f} features, {inv['size_mb'].sum():.0f} MB") layers features
crs
EPSG:27700 2 42506.0
EPSG:4326 24 28664.0
NaN 1 300.0
{'Polygon': 13, 'Point': 8, 'LineString': 4, 'MultiPolygon': 1, 'raster': 1}
path flags
osm_places_no_prj/places.shp no .prj; no CRS
total: 27 layers, 71,470 features, 121 MB
Two CRSs and one layer with none in a small delivery is a finding you want on day one, not on the day the spatial join returns nothing. Note that the raster has no features — its size is in width, height and bands — so the feature total is vectors only.
Inventorying a Rutland data delivery
The data used on this page
The setup assembles a folder shaped like a real delivery, from real downloads for Rutland:
osm/— Geofabrik's OpenStreetMap shapefile extract for Rutland of 1 January 2026, unpacked: 18 shapefiles, each with its sidecars.boundaries/rutland_wards.geojson— the 15 electoral wards from the ONS boundary service.parcels/— HM Land Registry's INSPIRE parcels for Rutland, both as the GML the Land Registry publishes and as a GeoJSON conversion of it.terrain/— the Copernicus GLO-30 elevation tile covering Rutland (52°N, 1°W), a Cloud-Optimised GeoTIFF.atlas.gpkg— a GeoPackage with three of the OSM layers.
Two defects are added deliberately, so the checks have something to find: boundaries/rutland_wards_FINAL.geojson is a byte-for-byte copy of the wards file under another name, and osm_places_no_prj/ holds the places shapefile without its .prj. Everything else is as downloaded.
import hashlib
import io
import os
import shutil
import time
import zipfile
from datetime import datetime, timezone
from pathlib import Path
import geopandas as gpd
import pandas as pd
import pyogrio
import rasterio
import requests
pyogrio.set_gdal_config_options({"GML_DOWNLOAD_SCHEMA": "NO"}) # the GML's schema URL is unreachable
root = Path("delivery")
Path("out").mkdir(exist_ok=True)
def fetch(url, **kwargs):
reply = requests.get(url, timeout=600, **kwargs)
reply.raise_for_status()
return reply.content
if not (root / "atlas.gpkg").exists():
for folder in ["osm", "boundaries", "parcels", "terrain", "osm_places_no_prj"]:
(root / folder).mkdir(parents=True, exist_ok=True)
zipfile.ZipFile(io.BytesIO(fetch("https://download.geofabrik.de/europe/united-kingdom/england/"
"rutland-260101-free.shp.zip"))).extractall(root / "osm")
(root / "boundaries/rutland_wards.geojson").write_bytes(fetch(
"https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/"
"Wards_December_2024_Boundaries_UK_BGC/FeatureServer/0/query",
params={"where": "LAD24NM='Rutland'", "outFields": "WD24CD,WD24NM", "outSR": 4326, "f": "geojson"}))
shutil.copy(root / "boundaries/rutland_wards.geojson", root / "boundaries/rutland_wards_FINAL.geojson")
inspire = requests.Session().get("https://use-land-property-data.service.gov.uk/datasets/inspire/download/"
"Rutland.zip", timeout=600)
inspire.raise_for_status()
zipfile.ZipFile(io.BytesIO(inspire.content)).extract("Land_Registry_Cadastral_Parcels.gml", root / "parcels")
gpd.read_file(root / "parcels/Land_Registry_Cadastral_Parcels.gml").to_file(root / "parcels/rutland_parcels.geojson")
(root / "terrain/copernicus_dem_N52_W001.tif").write_bytes(fetch(
"https://copernicus-dem-30m.s3.amazonaws.com/Copernicus_DSM_COG_10_N52_00_W001_00_DEM/"
"Copernicus_DSM_COG_10_N52_00_W001_00_DEM.tif"))
for ext in ["shp", "shx", "dbf", "cpg"]:
shutil.copy(root / f"osm/gis_osm_places_free_1.{ext}", root / f"osm_places_no_prj/places.{ext}")
for layer in ["landuse_a", "roads", "places"]:
gpd.read_file(root / f"osm/gis_osm_{layer}_free_1.shp").to_file(root / "atlas.gpkg", layer=layer)
files = [p for p in root.rglob("*") if p.is_file()]
print(len(files), "files,", f"{sum(p.stat().st_size for p in files) / 1e6:.0f} MB in", root)102 files, 118 MB in delivery
Example 1: the complete inventory script
#!/usr/bin/env python3
"""inventory.py — describe every GIS layer under a folder."""
from datetime import datetime, timezone
from pathlib import Path
import argparse, hashlib, sys
import pandas as pd
VECTOR_EXT = {".shp", ".gpkg", ".geojson", ".json", ".fgb", ".gml", ".kml", ".tab"}
RASTER_EXT = {".tif", ".tiff", ".img", ".asc", ".vrt", ".jp2"}
def quick_hash(path: Path, chunk=1 << 20) -> str:
h = hashlib.sha1(str(path.stat().st_size).encode())
with open(path, "rb") as fh:
h.update(fh.read(chunk))
return h.hexdigest()[:16]
def describe(path: Path) -> list[dict]:
import pyogrio, rasterio
stat = path.stat()
base = {
"path": str(path), "name": path.name, "ext": path.suffix.lower(),
"size_mb": round(stat.st_size / 1e6, 3),
"modified": datetime.fromtimestamp(
stat.st_mtime, timezone.utc).isoformat(timespec="seconds"),
"hash": quick_hash(path),
}
rows = []
try:
if path.suffix.lower() in VECTOR_EXT:
for layer_name, _ in pyogrio.list_layers(path):
info = pyogrio.read_info(path, layer=layer_name)
rows.append({**base, "kind": "vector", "layer": layer_name,
"features": int(info["features"]),
"geometry_type": info["geometry_type"],
"crs": info["crs"], "field_count": len(info["fields"]),
"fields": ",".join(info["fields"][:12]), "error": ""})
else:
with rasterio.open(path) as src:
rows.append({**base, "kind": "raster", "layer": path.stem,
"features": None, # cells are not features
"geometry_type": f"raster {src.width}x{src.height}x{src.count}",
"crs": src.crs.to_string() if src.crs else None,
"field_count": src.count, "fields": src.dtypes[0], "error": ""})
except Exception as exc:
rows.append({**base, "kind": "unknown", "layer": None, "features": None,
"geometry_type": None, "crs": None, "field_count": None,
"fields": "", "error": f"{type(exc).__name__}: {exc}"})
return rows
def main() -> int:
ap = argparse.ArgumentParser(description="Inventory a folder of GIS data")
ap.add_argument("root", type=Path)
ap.add_argument("-o", "--out", type=Path, default=Path("inventory.csv"))
args = ap.parse_args()
if not args.root.is_dir():
print(f"not a directory: {args.root}", file=sys.stderr)
return 2
files = [p for p in sorted(args.root.rglob("*"))
if p.is_file() and p.suffix.lower() in VECTOR_EXT | RASTER_EXT
and not p.name.startswith((".", "~", "$"))]
print(f"scanning {len(files)} files…", flush=True)
rows = []
for i, path in enumerate(files, start=1):
rows.extend(describe(path))
if i % 50 == 0:
print(f" [{i}/{len(files)}]", flush=True)
inv = pd.DataFrame(rows)
args.out.parent.mkdir(parents=True, exist_ok=True)
inv.to_csv(args.out, index=False)
print(f"\n{len(inv)} layers in {len(files)} files → {args.out}")
print(f"features : {inv['features'].sum():,.0f}")
print(f"size : {inv['size_mb'].sum()/1000:.2f} GB")
print(f"errors : {(inv['error'] != '').sum()}")
print("\nCRSs:")
print(inv["crs"].value_counts(dropna=False).head(10).to_string())
return 0
if __name__ == "__main__":
raise SystemExit(main())Run from a terminal it is python inventory.py delivery -o out/inventory_script.csv; here, the same call from Python:
import sys
sys.argv = ["inventory.py", "delivery", "-o", "out/inventory_script.csv"]
main()scanning 25 files…
27 layers in 25 files → out/inventory_script.csv
features : 71,470
size : 0.12 GB
errors : 0
CRSs:
crs
EPSG:4326 24
EPSG:27700 2
NaN 1
Example 2: find duplicate layers
import pandas as pd
inv = pd.read_csv("out/inventory_script.csv")
files = inv.drop_duplicates("path") # one row per file, not per layer
by_hash = files[files["hash"].duplicated(keep=False)].sort_values("hash")
print(f"{by_hash['hash'].nunique()} groups of byte-identical files")
print(by_hash[["hash", "path", "size_mb"]].head(12).to_string(index=False))
# same layer content under different names: same count, geometry, fields
sig = ["features", "geometry_type", "field_count", "fields"]
dupes = inv[inv.duplicated(sig, keep=False) & inv["features"].gt(0)].sort_values(sig)
print(f"\n{len(dupes)} layers share a content signature")
print(dupes[["path", "layer", "features"]].to_string(index=False))2 groups of byte-identical files
hash path size_mb
6c2fd33cb7df80f5 delivery/boundaries/rutland_wards.geojson 0.094
6c2fd33cb7df80f5 delivery/boundaries/rutland_wards_FINAL.geojson 0.094
f11a78b0f299190f delivery/osm/gis_osm_places_free_1.shp 0.009
f11a78b0f299190f delivery/osm_places_no_prj/places.shp 0.009
9 layers share a content signature
path layer features
delivery/boundaries/rutland_wards.geojson rutland_wards 15.0
delivery/boundaries/rutland_wards_FINAL.geojson rutland_wards_FINAL 15.0
delivery/atlas.gpkg places 300.0
delivery/osm/gis_osm_places_free_1.shp gis_osm_places_free_1 300.0
delivery/osm_places_no_prj/places.shp places 300.0
delivery/atlas.gpkg roads 7292.0
delivery/osm/gis_osm_roads_free_1.shp gis_osm_roads_free_1 7292.0
delivery/parcels/Land_Registry_Cadastral_Parcels.gml PREDEFINED 21253.0
delivery/parcels/rutland_parcels.geojson rutland_parcels 21253.0
The partial hash finds the wards file and its _FINAL copy, and the places .shp that the no-.prj folder copied byte for byte. The content signature finds re-exports that no byte comparison would: two of the three layers in atlas.gpkg match the OSM shapefiles they were made from, and the parcels GeoJSON matches the GML. The third GeoPackage layer, land use, escaped because it was written as MultiPolygon — a signature is a hint, not a proof.
Example 3: compare two deliveries
With two manifests of the same delivery taken a month apart (the file names here are placeholders):
import pandas as pd
old = pd.read_csv("data/out/inventory_2025.csv").set_index(["name", "layer"])
new = pd.read_csv("data/out/inventory_2026.csv").set_index(["name", "layer"])
added = new.index.difference(old.index)
removed = old.index.difference(new.index)
common = new.index.intersection(old.index)
print(f"added: {len(added)}, removed: {len(removed)}, common: {len(common)}")
changed = new.loc[common, "features"] - old.loc[common, "features"]
moved = changed[changed != 0].sort_values(key=abs, ascending=False)
print("\nbiggest feature-count changes:")
print(moved.head(10).to_string())This is how you answer "what actually changed in this month's delivery?" without opening anything.
Example 4: turn the inventory into a work queue
import pandas as pd
inv = pd.read_csv("out/inventory_script.csv")
todo = inv[inv["error"].isna() & (inv["features"] > 0)].copy() # "" is read back from CSV as NaN
todo["needs_reprojection"] = todo["crs"] != "EPSG:27700" # the target: British National Grid
todo["big"] = todo["size_mb"] > 500
print(todo.groupby(["needs_reprojection", "big"]).size())
todo.sort_values("size_mb", ascending=False).to_csv("out/work_queue.csv", index=False)needs_reprojection big
False False 2
True False 24
dtype: int64
Here 23 of the 26 usable layers are in WGS 84 and would need reprojecting to British National Grid before analysis. Processing the largest files first surfaces the expensive failures early, when you still have time to react.
Why a derived inventory beats a README
An inventory is the cheapest form of data documentation, and its value comes from being derived rather than written. Nobody updates a README; a script re-run against the folder is correct by construction, and it can be run again the moment the data changes.
What makes it fast is the distinction between metadata and data. Most GIS formats store a header describing the layer — feature count, geometry type, CRS, field schema, extent — and GDAL exposes it without reading a single feature or pixel. For those, pyogrio.read_info() and rasterio.open() are header reads, and scan time scales with the number of files, not their size. Text formats are the exception: GeoJSON, and GML until its .gfs exists, have to be parsed to be counted, so a folder of large GeoJSON inventories at the speed of reading it.
The unit of the inventory is the layer, not the file, and that distinction matters. A shapefile is one layer spread over five files; a GeoPackage is one file holding any number of layers; a File Geodatabase is a directory of many. Rows keyed by (path, layer) handle all three, and they make the feature counts add up.
Quality flags are what turn a description into something you act on. A missing .prj, a null CRS, a zero-feature layer and a zero-extent bounding box are all detectable from metadata alone, and each is a specific problem with a specific owner. Finding them on day one, in a table you can send to the data supplier, is dramatically cheaper than discovering them one at a time over a fortnight of processing.
Finally, keep the manifests. Two inventories a month apart let you diff a delivery: what appeared, what vanished, which layers changed size, whether a CRS quietly changed. That diff is the closest thing to change control most GIS deliveries ever get.
Geodatabases, zips and sidecars
- A
.gdbis a directory:rglob("*")will not yield it as a file. Detect directories ending in.gdband pass the folder path topyogrio.list_layers(). - Zipped data: GDAL can read
/vsizip/paths. Inventory archives without unpacking by prefixing the path. .shpsidecars are not layers: Skip.shx,.dbf,.prj,.cpgand.qixin discovery, or every shapefile appears five times.- Feature counts can be unknown: Some drivers return
-1rather than scan. Treat negative counts as missing, not as zero, or passforce_feature_count=Trueand accept the scan. - Rasters have cells, not features: Keep width, height and bands in their own columns; adding cells to a feature total (an easy slip) makes the total meaningless.
- CRS strings vary in form: The same CRS may appear as
EPSG:27700, an authority-less WKT, or a PROJ string. Normalise withpyproj.CRS.from_user_input(...).to_string()before grouping. - Network shares are slow to walk: Cache the inventory and re-scan on a schedule rather than at the start of every job.
- Hashing whole files is expensive: A partial hash of the head, tail and size is enough to find duplicates in practice.
Internal links
- How to Batch Process a Folder of GIS Files in Python: The Complete Workflow
- How to Process GIS Files in Nested Subfolders and Mirror the Output Tree
- How to Standardise and Repair CRS Across a Folder of Files in Python
- Python glob Is Not Finding All My Shapefiles: How to Fix It
- How to Validate a GeoDataFrame Against a Schema Before Analysis
- How to Record Run Metadata and Data Lineage in a GIS Pipeline
FAQ
How do I inventory a GeoPackage with many layers?
List them first with pyogrio.list_layers(path) and emit one inventory row per layer. A file-per-row inventory under-reports multi-layer containers badly.
What should I record about each layer?
Path, layer name, feature count, geometry type, CRS, field names and count, extent, file size and modification time — plus quality flags for missing CRS, empty layers and missing sidecars.
How do I find duplicate datasets?
Hash the first and last megabyte plus the file size to catch byte-identical copies, and separately group by a content signature (feature count, geometry type, field list) to catch re-exports under new names.
Can I inventory rasters the same way?
Yes. rasterio.open() reads the profile without decoding pixels, giving width, height, band count, dtype, CRS, pixel size, nodata and bounds.
How often should I re-run it?
Whenever a delivery arrives, and on a schedule for shared drives. Keeping the old manifests lets you diff deliveries and see exactly what changed.
Should the inventory live in the repo or in the data folder?
Write it next to the data so it travels with the delivery, and keep a copy in the run's output folder so each pipeline run records the state of its inputs.