⚙️ How-To

How to Process GIS Files in Nested Subfolders and Mirror the Output Tree

The data did not arrive as one tidy folder. It came organised by region, then by survey date, then by project — uk/2024/roads.shp, fr/2025/rivers.shp, dozens of leaf folders deep. A flat glob("*.shp") finds nothing at the top level, and even when you do reach the files, you do not want all the output dumped into one folder where roads.shp from three different regions collide. This guide shows you how to walk the whole tree, process every file, and write the results into an output directory that mirrors the input structure exactly — same folders, same layout, one clean file per source file.

Problem statement

You have GIS data spread across a nested directory tree and you are hitting some mix of these symptoms:

  • A flat glob finds nothing — your files live in region/date/project/ subfolders, so Path("data").glob("*.shp") returns an empty list because it only looks one level down.
  • The folder structure carries meaning — the path uk/2024/roads.shp encodes region and year, and you need to preserve that when you write output, not flatten it away.
  • Name collisions when you flatten — several subfolders each contain a roads.shp, so dumping them into one output folder overwrites all but the last.
  • The job reads its own output — you write results back under the input tree, the next run globs them up as if they were inputs, and the batch grows on every pass.
  • Shapefile sidecars confuse the walk — you are unsure whether to copy the .dbf, .shx, and .prj files or let GeoPandas handle them.

The goal: one repeatable script that recurses into every subfolder, processes each file, and rebuilds the identical folder tree under a separate output directory — with no collisions and no feedback loop.

Quick answer

Walk the tree with Path.rglob("*.shp"), and for each file compute its path relative to the source root with relative_to(). Join that relative path onto the output directory, swap the extension with with_suffix(), create the parent folders with mkdir(parents=True, exist_ok=True), then read, process, and write. Keep the output directory outside the input tree so the recursion never picks up its own results.

A nested input folder tree mirrored to an identical output tree by processing every file.
Recurse the input tree, then rebuild the same folders under the output directory, one file for one file.
import geopandas as gpd
from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/clean")

for path in sorted(src_dir.rglob("*.shp")):
    rel = path.relative_to(src_dir)          # uk/2024/roads.shp
    dest = out_dir / rel.with_suffix(".gpkg")  # data/clean/uk/2024/roads.gpkg
    dest.parent.mkdir(parents=True, exist_ok=True)

    gdf = gpd.read_file(path)
    gdf = gdf.to_crs("EPSG:4326")            # your processing goes here
    gdf.to_file(dest, driver="GPKG")
    print(f"{rel} -> {dest.relative_to(out_dir)}")

That is the whole pattern. The rest of this article explains each moving part — recursion, the relative-path trick, extension swapping, sidecars, and the feedback-loop trap — so you can adapt it without breaking the mirror.

Step-by-step solution

Recurse the tree instead of globbing one level

The single change that makes this work is rglob in place of glob. Where glob("*.shp") matches only the immediate folder, rglob("*.shp") (recursive glob) descends into every subfolder at any depth and yields every match it finds.

from pathlib import Path

src_dir = Path("data/raw")

for path in sorted(src_dir.rglob("*.shp")):
    print(path)
# data/raw/uk/2024/roads.shp
# data/raw/uk/2025/sites.shp
# data/raw/fr/2024/rivers.shp

sorted() gives you a stable, predictable order — useful for reproducible runs and readable logs. rglob returns a generator, so wrap it in sorted() or list() when you want to count or order the results.

Derive the mirrored output path

Each discovered path is absolute-ish and rooted at src_dir. To rebuild it under out_dir, strip the source root with relative_to() to get just the inner part of the path, then join that onto the output directory.

from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/clean")

path = src_dir / "uk/2024/roads.shp"
rel = path.relative_to(src_dir)          # uk/2024/roads.shp
dest = out_dir / rel                      # data/clean/uk/2024/roads.shp

relative_to(src_dir) is the key: it returns the portion of the path below the root, which is exactly the sub-tree you want to reproduce. Everything else is a plain path join.

Swap the extension with with_suffix()

If you are converting formats — shapefile in, GeoPackage out — change the extension on the relative path with with_suffix() before you join it. This keeps the folder structure identical while renaming only the leaf file.

rel = path.relative_to(src_dir)              # uk/2024/roads.shp
dest = out_dir / rel.with_suffix(".gpkg")    # data/clean/uk/2024/roads.gpkg

with_suffix(".gpkg") replaces the last extension and leaves the parent folders untouched. If you are writing the same format back out, skip this call and keep the original name.

Create the parent folders before writing

The mirrored folders do not exist yet on the first run. GeoPandas will not create them for you, so make the destination's parent directory before every write with mkdir(parents=True, exist_ok=True).

dest.parent.mkdir(parents=True, exist_ok=True)
gdf.to_file(dest, driver="GPKG")

parents=True creates every missing intermediate folder (data/clean/uk/2024/ in one call), and exist_ok=True makes it a no-op when the folder is already there — so it is safe to call on every iteration without a pre-check.

Keep the output tree out of the input tree

If out_dir sits inside src_dir, the next run's rglob will find your previous outputs and process them as if they were fresh inputs. Put the output directory somewhere the recursion cannot reach — a sibling of the input, not a child — or filter it out explicitly.

src_dir = Path("data/raw")     # inputs
out_dir = Path("data/clean")   # sibling, NOT data/raw/clean

# If they must share a parent you recurse, skip the output branch:
for path in src_dir.rglob("*.shp"):
    if out_dir in path.parents:
        continue
    ...

The cleanest fix is layout: keep inputs and outputs in separate top-level folders. The if out_dir in path.parents guard is the fallback when you cannot change the directory structure.

Code examples

Example 1: List every shapefile in the tree with its relative path

Before processing anything, walk the tree and print each file alongside the relative path you will mirror. This is the fastest way to confirm the recursion sees what you expect.

from pathlib import Path

src_dir = Path("data/raw")

for path in sorted(src_dir.rglob("*.shp")):
    rel = path.relative_to(src_dir)
    print(f"{rel}   (depth {len(rel.parts) - 1})")
# uk/2024/roads.shp   (depth 2)
# uk/2025/sites.shp   (depth 2)
# fr/2024/rivers.shp  (depth 2)

rel.parts splits the relative path into its components, so len(rel.parts) - 1 tells you how deep each file sits — handy for spotting a stray file at an unexpected level.

Example 2: The mirror loop — relative_to, dest, mkdir, process, write

This is the core pattern end to end, with per-file error handling so one bad file does not stop the tree walk.

import geopandas as gpd
from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/clean")

def process(gdf):
    gdf = gdf.copy()
    if gdf.crs is None:
        raise ValueError("no CRS defined")
    return gdf.to_crs("EPSG:4326")

failed = []
for path in sorted(src_dir.rglob("*.shp")):
    rel = path.relative_to(src_dir)
    dest = out_dir / rel.with_suffix(".gpkg")
    dest.parent.mkdir(parents=True, exist_ok=True)

    try:
        gdf = gpd.read_file(path)
        process(gdf).to_file(dest, driver="GPKG")
        print(f"OK   {rel}")
    except Exception as exc:
        failed.append((str(rel), str(exc)))
        print(f"FAIL {rel}: {exc}")

print(f"\nDone. {len(failed)} failed.")

Example 3: Recurse across multiple extensions

Real trees mix formats. Recurse each pattern in turn and chain the results, so one loop handles shapefiles, GeoJSON, and GeoPackage inputs together.

import geopandas as gpd
from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/clean")
PATTERNS = ("*.shp", "*.geojson", "*.gpkg")

files = sorted(p for pat in PATTERNS for p in src_dir.rglob(pat))

for path in files:
    rel = path.relative_to(src_dir)
    dest = out_dir / rel.with_suffix(".gpkg")
    dest.parent.mkdir(parents=True, exist_ok=True)
    gdf = gpd.read_file(path).to_crs("EPSG:4326")
    gdf.to_file(dest, driver="GPKG")

The generator expression for pat in PATTERNS for p in src_dir.rglob(pat) runs each recursive glob and flattens the results into one sorted list.

Example 4: Guard against reading files already in the output folder

When you cannot keep the output outside the input tree, filter it out inside the loop so the batch never eats its own results.

import geopandas as gpd
from pathlib import Path

root = Path("data")            # holds both raw/ and clean/
src_dir = root / "raw"
out_dir = root / "clean"

for path in sorted(src_dir.rglob("*.shp")):
    # Belt and braces: never process anything under the output dir.
    if out_dir in path.parents:
        continue

    rel = path.relative_to(src_dir)
    dest = out_dir / rel.with_suffix(".gpkg")
    if dest.exists():
        continue               # already done — skip on a re-run

    dest.parent.mkdir(parents=True, exist_ok=True)
    gpd.read_file(path).to_crs("EPSG:4326").to_file(dest, driver="GPKG")

The dest.exists() check also makes the walk resumable: interrupt it, run it again, and it picks up where it left off.

Example 5: Flatten into one folder without name collisions

Sometimes you want everything in a single output folder rather than a mirror. Derive a unique filename from the relative path so uk/2024/roads.shp and fr/2024/roads.shp do not overwrite each other.

import geopandas as gpd
from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/flat")
out_dir.mkdir(parents=True, exist_ok=True)

for path in sorted(src_dir.rglob("*.shp")):
    rel = path.relative_to(src_dir).with_suffix(".gpkg")
    flat_name = "__".join(rel.parts)         # uk__2024__roads.gpkg
    dest = out_dir / flat_name
    gpd.read_file(path).to_crs("EPSG:4326").to_file(dest, driver="GPKG")

Joining rel.parts with a separator folds the folder hierarchy into the filename, so every output name is unique and still traceable back to its source path.

Explanation

Why rglob and not glob

Path.glob(pattern) matches only within the directory you call it on; Path.rglob(pattern) is shorthand for glob("**/" + pattern), which means "match this pattern in this folder and every folder beneath it, at any depth." That recursive ** is the whole difference. Because it returns a lazy generator, iterating it once consumes it — call sorted() or list() if you need to reuse, count, or order the matches. rglob follows the directory tree as it is laid out on disk; it does not follow symlinks into cycles, so a well-formed tree walks cleanly.

How relative_to() rebuilds the tree

relative_to() answers a precise question: given a full path and a root it lives under, what is the path from the root to the file? For data/raw/uk/2024/roads.shp relative to data/raw, the answer is uk/2024/roads.shp — exactly the sub-tree you want to recreate elsewhere. Joining that onto out_dir with the / operator reproduces the structure verbatim. The one rule: the path must genuinely sit under the root you pass, or relative_to() raises a ValueError. Since every path came from src_dir.rglob(...), that invariant holds automatically.

Why mkdir(parents=True, exist_ok=True) on every iteration

The output folders do not exist until you create them, and different files need different parents (uk/2024/, fr/2024/, and so on). parents=True builds the entire missing chain in one call rather than one level at a time, and exist_ok=True turns a "directory already exists" error into a silent no-op. Together they make the call idempotent — safe to run unconditionally before every write, with no need to check first whether the folder is there.

Edge cases or notes

Shapefile sidecars come along automatically

A shapefile is really four-plus files: .shp, .dbf, .shx, and usually .prj. You only ever rglob("*.shp") and only ever read the .shp — GeoPandas pulls in the sidecars for you from the same folder. Do not glob or copy the sidecars yourself. And because the recommended output is GeoPackage, a single self-contained file, the whole sidecar problem disappears on the way out.

Empty leaf folders and non-data files

A tree often contains readme.txt, .xml metadata, thumbnails, or empty folders. Because you match specific patterns (*.shp, *.geojson), rglob simply ignores everything else — non-matching files never enter the loop, and empty folders yield nothing. The mirrored output tree only ever contains folders that hold at least one processed file, which is usually what you want.

Flatten versus mirror is a real choice

Mirroring preserves the region/date/project meaning encoded in the paths and keeps outputs easy to diff against inputs. Flattening (Example 5) is simpler to hand to a tool that expects one folder, but you lose the hierarchy unless you fold it into the filename. Pick mirroring when the structure is meaningful and you may re-run incrementally; pick flattening when a downstream step wants a single directory.

The output-inside-input feedback loop

The nastiest bug here is silent: put data/raw/clean/ under data/raw/, and every run globs the previous run's outputs back in, doubling work and corrupting your mirror. Prefer a directory layout where inputs and outputs are siblings. When you cannot, guard with if out_dir in path.parents: continue as in Example 4. This is the single most common mistake with recursive batch jobs.

Combine the whole tree into one GeoPackage

Instead of a mirrored tree of files, you can collapse everything into a single GeoPackage with one layer per source file, keyed by the relative path so layer names stay unique.

import geopandas as gpd
from pathlib import Path

src_dir = Path("data/raw")
out_path = Path("data/clean/all.gpkg")
out_path.parent.mkdir(parents=True, exist_ok=True)

for path in sorted(src_dir.rglob("*.shp")):
    rel = path.relative_to(src_dir).with_suffix("")
    layer = "__".join(rel.parts)             # uk__2024__roads
    gdf = gpd.read_file(path).to_crs("EPSG:4326")
    gdf.to_file(out_path, layer=layer, driver="GPKG")

One GeoPackage with descriptive layer names keeps an entire tree in a single portable file, and the relative-path-derived layer name prevents collisions.

Very deep or very large trees

rglob streams matches lazily, so listing even a huge tree is cheap; the cost is in reading and writing each file. Process one file per iteration (as shown) so only one GeoDataFrame is resident at a time. If you have many independent files and spare cores, distribute the loop with concurrent.futures.ProcessPoolExecutor — the mirror pattern parallelises cleanly because each file's destination is computed independently.

FAQ

What is the difference between glob and rglob in pathlib?

glob(pattern) matches only files in the directory you call it on, one level deep. rglob(pattern) recurses into every subfolder at any depth — it is equivalent to glob("**/" + pattern). Use rglob whenever your data is organised into nested folders by region, date, or project.

How do I recreate the input folder structure in my output directory?

For each file, compute path.relative_to(src_dir) to get the inner sub-path, join it onto your output directory, and call dest.parent.mkdir(parents=True, exist_ok=True) before writing. The relative path carries the folder structure, so joining it under the output root reproduces the tree exactly.

How do I change the file extension while keeping the folder path?

Use with_suffix() on the relative path: rel.with_suffix(".gpkg") turns uk/2024/roads.shp into uk/2024/roads.gpkg, replacing only the extension and leaving the parent folders untouched. Join the result onto your output directory as usual.

How do I stop the batch from processing its own output files?

Keep the output directory outside the input tree — make them siblings, not parent and child — so rglob never reaches your results. If they must share a parent you recurse over, add if out_dir in path.parents: continue inside the loop to skip anything under the output folder.

Do I need to copy shapefile sidecar files like .dbf and .shx?

No. Only ever glob and read the .shp; GeoPandas loads the .dbf, .shx, and .prj sidecars automatically from the same folder. If you write GeoPackage output, the result is a single self-contained file and there are no sidecars to manage at all.

Should I mirror the tree or flatten everything into one folder?

Mirror when the folder structure carries meaning (region, date, project) and you want outputs that diff cleanly against inputs. Flatten when a downstream tool expects a single directory — but derive the filename from the relative path (for example "__".join(rel.parts)) so files with the same name in different subfolders do not overwrite each other.

How do I make the recursive batch resumable?

Check if dest.exists(): continue before processing each file. On a re-run, files whose mirrored output already exists are skipped, so an interrupted job resumes from where it stopped. See the resumable batch job guide for a more complete approach with a manifest.

Can I put the whole tree into one GeoPackage instead of many files?

Yes. Write each file to the same .gpkg with a distinct layer= name derived from its relative path, such as "__".join(rel.with_suffix("").parts). This keeps an entire directory tree in one portable file, with unique layer names that avoid collisions between identically named files in different folders.