How to Build an Inventory of a GIS Data Folder in Python

Problem statement

Someone hands you a drive: \\server\projects\2026_delivery. Four hundred files, nested folders, a mixture of shapefiles, GeoPackages, GeoJSON and rasters, no documentation. Before you can process any of it you need to answer basic questions:

how many layers are there, and how many features in total?
what CRSs are in use — is it one, or eleven?
which files are broken, empty, or missing their sidecars?
which are duplicates of each other under different names?
what geometry types, and what attribute schemas?

Opening them one by one in QGIS takes a day. An inventory script takes a minute to write and seconds to run, and it produces a table you can sort, filter, share and re-run when the next delivery lands.

The trick is doing it cheaply: reading every feature of every file to count them is slow and unnecessary. Metadata reads answer most questions without touching the geometry.

Quick answer

Walk the tree, read metadata only, and write a table:

find candidate files by extension, skipping sidecars and temp files
read metadata with pyogrio.read_info() — feature count, CRS, geometry type, fields
catch per-file errors so one bad file cannot end the scan
add file-level facts: size, modification time, checksum
write a CSV or Parquet manifest, and summarise the CRSs and types

from pathlib import Path
import pandas as pd
import pyogrio

VECTOR_EXT = {".shp", ".gpkg", ".geojson", ".json", ".fgb", ".gml", ".kml", ".tab"}

rows = []
for path in sorted(Path("data/raw").rglob("*")):
    if not path.is_file() or path.suffix.lower() not in VECTOR_EXT:
        continue
    try:
        for layer in pyogrio.list_layers(path):
            name = layer[0]
            info = pyogrio.read_info(path, layer=name)
            rows.append({
                "file": str(path), "layer": name,
                "features": info["features"], "geometry": info["geometry_type"],
                "crs": info["crs"], "fields": len(info["fields"]),
                "size_mb": round(path.stat().st_size / 1e6, 2),
            })
    except Exception as exc:
        rows.append({"file": str(path), "layer": None, "error": f"{type(exc).__name__}: {exc}"})

inv = pd.DataFrame(rows)
inv.to_csv("data/out/inventory.csv", index=False)
print(inv.groupby("crs", dropna=False)["features"].agg(["count", "sum"]))

read_info() reads the header, not the data, so a 4 GB GeoPackage answers as fast as a 40 KB one.

What an inventory records

Grid of inventory fields grouped into file facts, layer facts and quality flags. — Three groups of columns — file, layer, and the flags that make the table actionable.

Step-by-step solution

Six stages, all metadata-only until the optional deep checks at the end.

Discover the real data files

A folder tree contains far more than data: sidecars, lock files, cloud placeholders, and your own outputs.

from pathlib import Path

VECTOR_EXT = {".shp", ".gpkg", ".geojson", ".json", ".fgb", ".gml", ".kml", ".tab", ".mif"}
RASTER_EXT = {".tif", ".tiff", ".img", ".asc", ".vrt", ".jp2", ".nc"}
SKIP_PARTS = {"__pycache__", ".git", ".ipynb_checkpoints", "$RECYCLE.BIN"}

def candidates(root: Path):
    for p in sorted(root.rglob("*")):
        if not p.is_file():
            continue
        if p.name.startswith((".", "~", "$")):
            continue
        if any(part in SKIP_PARTS for part in p.parts):
            continue
        ext = p.suffix.lower()
        if ext in VECTOR_EXT:
            yield p, "vector"
        elif ext in RASTER_EXT:
            yield p, "raster"

root = Path("data/raw")
found = list(candidates(root))
print(f"{len(found)} candidate files "
      f"({sum(1 for _, k in found if k == 'vector')} vector, "
      f"{sum(1 for _, k in found if k == 'raster')} raster)")

Note that .gdb is a directory, and a GeoPackage may hold many layers — so files and layers are not the same thing, which is why the inventory has a row per layer.

Read layer metadata without loading features

import pyogrio

def vector_rows(path):
    out = []
    for layer_name, geom_type in pyogrio.list_layers(path):
        info = pyogrio.read_info(path, layer=layer_name)
        out.append({
            "layer": layer_name,
            "features": int(info["features"]),
            "geometry_type": info["geometry_type"] or geom_type,
            "crs": info["crs"],
            "field_count": len(info["fields"]),
            "fields": ",".join(info["fields"][:12]),
            "bounds": [round(v, 4) for v in info["total_bounds"]]
                      if info.get("total_bounds") is not None else None,
        })
    return out

pyogrio.read_info returns the feature count, CRS as a WKT/authority string, geometry type, field names and dtypes, and the layer extent — everything the inventory needs, from the header.

With fiona instead:

import fiona

for layer_name in fiona.listlayers(path):
    with fiona.open(path, layer=layer_name) as src:
        info = {"features": len(src), "crs": src.crs_wkt,
                "geometry_type": src.schema["geometry"],
                "fields": list(src.schema["properties"])}

Add raster metadata

import rasterio

def raster_row(path):
    with rasterio.open(path) as src:
        return {
            "layer": path.stem,
            "geometry_type": "raster",
            "crs": src.crs.to_string() if src.crs else None,
            "width": src.width, "height": src.height, "bands": src.count,
            "dtype": src.dtypes[0],
            "pixel_size": round(abs(src.transform.a), 6),
            "nodata": src.nodata,
            "bounds": [round(v, 4) for v in src.bounds],
        }

Reading a raster's profile is likewise a header operation — no pixels are decoded.

Record the file-level facts

from datetime import datetime, timezone
import hashlib

def file_row(path):
    stat = path.stat()
    return {
        "path": str(path),
        "name": path.name,
        "ext": path.suffix.lower(),
        "size_mb": round(stat.st_size / 1e6, 3),
        "modified": datetime.fromtimestamp(
            stat.st_mtime, timezone.utc).isoformat(timespec="seconds"),
    }

def quick_hash(path: Path, chunk=1 << 20) -> str:
    """Hash the first and last MB plus the size — fast, good enough to spot duplicates."""
    h = hashlib.sha1(str(path.stat().st_size).encode())
    with open(path, "rb") as fh:
        h.update(fh.read(chunk))
        if path.stat().st_size > 2 * chunk:
            fh.seek(-chunk, 2)
            h.update(fh.read(chunk))
    return h.hexdigest()[:16]

A partial hash finds the copy-of-a-copy duplicates that plague delivery folders, without reading terabytes.

Flag the problems you can detect cheaply

REQUIRED_SIDECARS = {".shp": (".shx", ".dbf")}

def quality_flags(path: Path, row: dict) -> list[str]:
    flags = []
    for ext in REQUIRED_SIDECARS.get(path.suffix.lower(), ()):
        if not path.with_suffix(ext).exists():
            flags.append(f"missing{ext}")
    if path.suffix.lower() == ".shp" and not path.with_suffix(".prj").exists():
        flags.append("no .prj")
    if row.get("crs") in (None, ""):
        flags.append("no CRS")
    if row.get("features") == 0:
        flags.append("empty layer")
    if row.get("features", 0) and row.get("bounds"):
        minx, miny, maxx, maxy = row["bounds"]
        if minx == maxx and miny == maxy:
            flags.append("zero-extent")
    return flags

Flags turn the inventory from a description into a to-do list.

Write the manifest and summarise it

import pandas as pd

inv = pd.DataFrame(rows)
inv.to_csv("data/out/inventory.csv", index=False)
inv.to_parquet("data/out/inventory.parquet")

print("── by CRS")
by_crs = inv.groupby("crs", dropna=False).agg(layers=("layer", "size"),
                                              features=("features", "sum"))
print(by_crs.sort_values("features", ascending=False))
print("\n── by geometry type")
print(inv["geometry_type"].value_counts(dropna=False))
print("\n── problems")
flagged = inv.loc[inv["flags"].astype(bool), ["path", "layer", "flags"]]
print(flagged.head(20).to_string(index=False))
print(f"\ntotal: {len(inv)} layers, {inv['features'].sum():,.0f} features, "
      f"{inv['size_mb'].sum()/1000:.1f} GB")

Eleven CRSs in one delivery is a finding you want on day one, not on the day the spatial join returns nothing.

Code examples

Example 1: the complete inventory script

#!/usr/bin/env python3
"""inventory.py — describe every GIS layer under a folder."""
from datetime import datetime, timezone
from pathlib import Path
import argparse, hashlib, sys
import pandas as pd

VECTOR_EXT = {".shp", ".gpkg", ".geojson", ".json", ".fgb", ".gml", ".kml", ".tab"}
RASTER_EXT = {".tif", ".tiff", ".img", ".asc", ".vrt", ".jp2"}

def quick_hash(path: Path, chunk=1 << 20) -> str:
    h = hashlib.sha1(str(path.stat().st_size).encode())
    with open(path, "rb") as fh:
        h.update(fh.read(chunk))
    return h.hexdigest()[:16]

def describe(path: Path) -> list[dict]:
    import pyogrio, rasterio
    stat = path.stat()
    base = {
        "path": str(path), "name": path.name, "ext": path.suffix.lower(),
        "size_mb": round(stat.st_size / 1e6, 3),
        "modified": datetime.fromtimestamp(
            stat.st_mtime, timezone.utc).isoformat(timespec="seconds"),
        "hash": quick_hash(path),
    }
    rows = []
    try:
        if path.suffix.lower() in VECTOR_EXT:
            for layer_name, _ in pyogrio.list_layers(path):
                info = pyogrio.read_info(path, layer=layer_name)
                rows.append({**base, "kind": "vector", "layer": layer_name,
                             "features": int(info["features"]),
                             "geometry_type": info["geometry_type"],
                             "crs": info["crs"], "field_count": len(info["fields"]),
                             "fields": ",".join(info["fields"][:12]), "error": ""})
        else:
            with rasterio.open(path) as src:
                rows.append({**base, "kind": "raster", "layer": path.stem,
                             "features": src.width * src.height,
                             "geometry_type": f"raster {src.width}x{src.height}x{src.count}",
                             "crs": src.crs.to_string() if src.crs else None,
                             "field_count": src.count, "fields": src.dtypes[0], "error": ""})
    except Exception as exc:
        rows.append({**base, "kind": "unknown", "layer": None, "features": None,
                     "geometry_type": None, "crs": None, "field_count": None,
                     "fields": "", "error": f"{type(exc).__name__}: {exc}"})
    return rows

def main() -> int:
    ap = argparse.ArgumentParser(description="Inventory a folder of GIS data")
    ap.add_argument("root", type=Path)
    ap.add_argument("-o", "--out", type=Path, default=Path("inventory.csv"))
    args = ap.parse_args()

    if not args.root.is_dir():
        print(f"not a directory: {args.root}", file=sys.stderr)
        return 2

    files = [p for p in sorted(args.root.rglob("*"))
             if p.is_file() and p.suffix.lower() in VECTOR_EXT | RASTER_EXT
             and not p.name.startswith((".", "~", "$"))]
    print(f"scanning {len(files)} files…", flush=True)

    rows = []
    for i, path in enumerate(files, start=1):
        rows.extend(describe(path))
        if i % 50 == 0:
            print(f"  [{i}/{len(files)}]", flush=True)

    inv = pd.DataFrame(rows)
    args.out.parent.mkdir(parents=True, exist_ok=True)
    inv.to_csv(args.out, index=False)

    print(f"\n{len(inv)} layers in {len(files)} files → {args.out}")
    print(f"features : {inv['features'].sum():,.0f}")
    print(f"size     : {inv['size_mb'].sum()/1000:.2f} GB")
    print(f"errors   : {(inv['error'] != '').sum()}")
    print("\nCRSs:")
    print(inv["crs"].value_counts(dropna=False).head(10).to_string())
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Example 2: find duplicate layers

import pandas as pd

inv = pd.read_csv("data/out/inventory.csv")

by_hash = inv[inv["hash"].duplicated(keep=False)].sort_values("hash")
print(f"{by_hash['hash'].nunique()} groups of byte-identical files")
print(by_hash[["hash", "path", "size_mb"]].head(12).to_string(index=False))

# same layer content under different names: same count, geometry, fields
sig = ["features", "geometry_type", "field_count", "fields"]
dupes = inv[inv.duplicated(sig, keep=False) & inv["features"].gt(0)].sort_values(sig)
print(f"\n{len(dupes)} layers share a content signature")

Example 3: compare two deliveries

import pandas as pd

old = pd.read_csv("data/out/inventory_2025.csv").set_index(["name", "layer"])
new = pd.read_csv("data/out/inventory_2026.csv").set_index(["name", "layer"])

added = new.index.difference(old.index)
removed = old.index.difference(new.index)
common = new.index.intersection(old.index)

print(f"added: {len(added)}, removed: {len(removed)}, common: {len(common)}")

changed = new.loc[common, "features"] - old.loc[common, "features"]
moved = changed[changed != 0].sort_values(key=abs, ascending=False)
print("\nbiggest feature-count changes:")
print(moved.head(10).to_string())

This is how you answer "what actually changed in this month's delivery?" without opening anything.

Example 4: turn the inventory into a work queue

import pandas as pd

inv = pd.read_csv("data/out/inventory.csv")

todo = inv[(inv["error"] == "") & (inv["features"] > 0)].copy()
todo["needs_reprojection"] = todo["crs"] != "EPSG:27700"
todo["big"] = todo["size_mb"] > 500

print(todo.groupby(["needs_reprojection", "big"]).size())
todo.sort_values("size_mb", ascending=False).to_csv("data/out/work_queue.csv", index=False)

Processing the largest files first surfaces the expensive failures early, when you still have time to react.

Explanation

An inventory is the cheapest form of data documentation, and its value comes from being derived rather than written. Nobody updates a README; a script re-run against the folder is correct by construction, and it can be run again the moment the data changes.

Panels contrasting opening files one by one against a derived manifest. — The same questions, answered in seconds and repeatably.

What makes it fast is the distinction between metadata and data. Every GIS format stores a header describing the layer — feature count, geometry type, CRS, field schema, extent — and both OGR and GDAL expose it without reading a single feature or pixel. pyogrio.read_info() and rasterio.open().profile are header reads, so scan time scales with the number of files, not with their size. A 200 GB folder inventories in about as long as a 2 GB one.

The unit of the inventory is the layer, not the file, and that distinction matters. A shapefile is one layer spread over five files; a GeoPackage is one file holding any number of layers; a File Geodatabase is a directory of many. Rows keyed by (path, layer) handle all three, and they make the feature counts add up.

Quality flags are what turn a description into something you act on. A missing .prj, a null CRS, a zero-feature layer and a zero-extent bounding box are all detectable from metadata alone, and each is a specific problem with a specific owner. Finding them on day one, in a table you can send to the data supplier, is dramatically cheaper than discovering them one at a time over a fortnight of processing.

Finally, keep the manifests. Two inventories a month apart let you diff a delivery: what appeared, what vanished, which layers changed size, whether a CRS quietly changed. That diff is the closest thing to change control most GIS deliveries ever get.

Edge cases or notes

A .gdb is a directory: rglob("*") will not yield it as a file. Detect directories ending in .gdb and pass the folder path to pyogrio.list_layers().
Zipped data: GDAL can read /vsizip/ paths. Inventory archives without unpacking by prefixing the path.
.shp sidecars are not layers: Skip .shx, .dbf, .prj, .cpg and .qix in discovery, or every shapefile appears five times.
Feature counts can be approximate: Some drivers return -1 for unknown. Treat negative counts as missing, not as zero.
CRS strings vary in form: The same CRS may appear as EPSG:27700, an authority-less WKT, or a PROJ string. Normalise with pyproj.CRS.from_user_input(...).to_string() before grouping.
Network shares are slow to walk: Cache the inventory and re-scan on a schedule rather than at the start of every job.
Hashing whole files is expensive: A partial hash of the head, tail and size is enough to find duplicates in practice.

Internal links

FAQ

How do I count features without loading the data?

pyogrio.read_info(path, layer=...)["features"] reads the layer header only. fiona.open(...) with len() does the same. Neither decodes geometry, so it is fast regardless of file size.

How do I inventory a GeoPackage with many layers?

List them first with pyogrio.list_layers(path) and emit one inventory row per layer. A file-per-row inventory under-reports multi-layer containers badly.

What should I record about each layer?

Path, layer name, feature count, geometry type, CRS, field names and count, extent, file size and modification time — plus quality flags for missing CRS, empty layers and missing sidecars.

How do I find duplicate datasets?

Hash the first and last megabyte plus the file size to catch byte-identical copies, and separately group by a content signature (feature count, geometry type, field list) to catch re-exports under new names.

Can I inventory rasters the same way?

Yes. rasterio.open() reads the profile without decoding pixels, giving width, height, band count, dtype, CRS, pixel size, nodata and bounds.

How often should I re-run it?

Whenever a delivery arrives, and on a schedule for shared drives. Keeping the old manifests lets you diff deliveries and see exactly what changed.

Should the inventory live in the repo or in the data folder?

Write it next to the data so it travels with the delivery, and keep a copy in the run's output folder so each pipeline run records the state of its inputs.

How to Build an Inventory of a GIS Data Folder in Python #

Problem statement #

Quick answer #

What an inventory records #

Step-by-step solution #

Discover the real data files #

Read layer metadata without loading features #

Add raster metadata #

Record the file-level facts #

Flag the problems you can detect cheaply #

Write the manifest and summarise it #

Code examples #

Example 1: the complete inventory script #

Example 2: find duplicate layers #

Example 3: compare two deliveries #

Example 4: turn the inventory into a work queue #

Explanation #

Edge cases or notes #

Internal links #

FAQ #

How do I count features without loading the data? #

How do I inventory a GeoPackage with many layers? #

What should I record about each layer? #

How do I find duplicate datasets? #

Can I inventory rasters the same way? #

How often should I re-run it? #

Should the inventory live in the repo or in the data folder? #