🔧 Fix Batch processing Batch Processing · step 24 of 26 Updated

Batch Output Files Keep Overwriting Each Other: How to Fix It

Problem statement

The batch run reports 240 files processed. The output folder contains one.

processed 240 files
$ ls data/out
output.gpkg

Or the count is right but the contents are wrong: roads.gpkg in the output holds the last region's roads, and 11 earlier regions have vanished. Nothing raised an error — every write succeeded. They just all wrote to the same place.

Where this comes from:

  • the output filename is a constant, built outside the loop
  • a recursive input walk flattens the tree, so north/roads.shp and south/roads.shp both become roads.gpkg
  • to_file() on an existing GeoPackage replaces the layer instead of adding one
  • the output name is derived from a column value that repeats
  • a parallel run has two workers writing the same path at the same time
  • the output directory is also the input directory, so the second pass consumes its own results

The symptom is silent because writing a file that already exists is a normal operation. Nothing in GeoPandas, Fiona, or the filesystem considers it an error, so the loss is only visible when you count the results.

Quick answer

To stop outputs from colliding:

  1. derive the output name from the input path inside the loop — never a fixed string
  2. preserve the input folder structure with relative_to() instead of flattening it
  3. add a discriminator (region, date, layer name) when stems can repeat
  4. check for an existing target before writing, and either skip, version, or fail
  5. write to a temporary name and rename on success so partial files are never mistaken for results
from pathlib import Path
import geopandas as gpd

SRC, OUT = Path("data/raw").resolve(), Path("data/out").resolve()

for shp in sorted(SRC.rglob("*.shp")):
    rel = shp.relative_to(SRC).with_suffix(".gpkg")   # north/roads.gpkg
    dest = OUT / rel
    dest.parent.mkdir(parents=True, exist_ok=True)
    if dest.exists():
        raise FileExistsError(f"refusing to overwrite {dest}")
    gpd.read_file(shp).to_file(dest, driver="GPKG")

The two lines that matter are relative_to() — which keeps north/ and south/ apart — and the exists() guard, which converts a silent overwrite into a loud failure.

How outputs collide

Three input files from different folders all mapping to one output filename.
Flattening a tree collapses distinct inputs onto one name — the last writer wins.

Step-by-step solution

Four naming strategies for batch outputs compared.
Four ways to build a unique output path, and what each one is good for.

Move the output path inside the loop

The classic bug is a path computed once, above the loop, and reused for every iteration.

# wrong: one target for every input
out_path = Path("data/out/output.gpkg")
for shp in files:
    gpd.read_file(shp).to_file(out_path, driver="GPKG")

# right: one target per input
for shp in files:
    out_path = Path("data/out") / f"{shp.stem}.gpkg"
    gpd.read_file(shp).to_file(out_path, driver="GPKG")

If you want everything in one file, that is a different operation — concatenate the frames and write once, or append layers explicitly (see below). Repeatedly overwriting is never the way to combine data.

Mirror the input tree instead of flattening it

Path.stem throws away the folder, which is exactly the information that made two same-named files distinct.

from pathlib import Path

SRC = Path("data/raw").resolve()
OUT = Path("data/out").resolve()

for shp in sorted(SRC.rglob("*.shp")):
    rel = shp.relative_to(SRC)                 # north/roads.shp
    dest = (OUT / rel).with_suffix(".gpkg")    # data/out/north/roads.gpkg
    dest.parent.mkdir(parents=True, exist_ok=True)

mkdir(parents=True, exist_ok=True) is required — writers do not create missing intermediate directories, and the resulting error is a confusing "No such file or directory" on a path you can see in the message.

Or flatten deliberately, with the folder in the name

Sometimes a flat output folder is genuinely easier to consume. Then encode the structure in the filename rather than discarding it.

rel = shp.relative_to(SRC).with_suffix("")     # north/roads
flat_name = "_".join(rel.parts) + ".gpkg"      # north_roads.gpkg
dest = OUT / flat_name

This keeps a single directory listing while guaranteeing uniqueness, and the origin of each file remains readable.

Detect collisions before writing anything

The cheapest fix is to compute every output path up front and check for duplicates. A collision found in the first second is far better than one found after two hours.

from collections import Counter

targets = {}
for shp in sorted(SRC.rglob("*.shp")):
    dest = (OUT / shp.relative_to(SRC)).with_suffix(".gpkg")
    targets.setdefault(dest, []).append(shp)

clashes = {d: srcs for d, srcs in targets.items() if len(srcs) > 1}
if clashes:
    for dest, srcs in clashes.items():
        print(f"! {dest.name} would be written by {len(srcs)} inputs:")
        for s in srcs:
            print(f"    {s}")
    raise SystemExit("output paths collide — fix the naming rule")

Understand what to_file() does to an existing GeoPackage

This one surprises people, because it is not a filesystem overwrite. Writing a GeoPackage that already exists replaces the layer of that name and leaves other layers intact — unless you ask for something else.

import geopandas as gpd

# replaces the layer "roads" inside atlas.gpkg, keeps other layers
gdf.to_file("atlas.gpkg", layer="roads", driver="GPKG")

# adds rows to an existing layer instead of replacing it
gdf.to_file("atlas.gpkg", layer="roads", driver="GPKG", mode="a")

So a loop that writes every region to atlas.gpkg with the same layer= name ends up with only the last region — but a loop that varies layer= produces a single, tidy multi-layer file. That is often exactly what you want.

for shp in sorted(SRC.rglob("*.shp")):
    layer = "_".join(shp.relative_to(SRC).with_suffix("").parts)
    gpd.read_file(shp).to_file(OUT / "atlas.gpkg", layer=layer, driver="GPKG")

Shapefiles have no such concept: one shapefile is one layer, so unique filenames are the only option there.

Write atomically so failures cannot corrupt a good file

If the process dies mid-write, you are left with a file that exists but is incomplete — and the next run will happily treat it as done.

from pathlib import Path
import geopandas as gpd

def write_atomic(gdf, dest: Path, driver="GPKG"):
    dest.parent.mkdir(parents=True, exist_ok=True)
    tmp = dest.with_name(dest.name + ".part")
    gdf.to_file(tmp, driver=driver)
    tmp.replace(dest)          # atomic on the same filesystem

Path.replace() is an atomic rename on POSIX and overwrites on Windows too, so a reader never observes a half-written result.

Decide the overwrite policy explicitly

Every batch needs one of three behaviours, chosen on purpose:

MODE = "skip"      # "skip" | "overwrite" | "version" | "fail"

if dest.exists():
    if MODE == "skip":
        continue
    if MODE == "fail":
        raise FileExistsError(dest)
    if MODE == "version":
        n = 1
        while dest.with_stem(f"{dest.stem}_{n}").exists():
            n += 1
        dest = dest.with_stem(f"{dest.stem}_{n}")
    # "overwrite" falls through and writes

skip doubles as a crude resume mechanism; fail is the right default while you are still developing the naming rule.

Code examples

Example 1: safe per-file conversion with a mirrored tree 

from pathlib import Path
import geopandas as gpd

SRC = Path("data/raw").resolve()
OUT = Path("data/out").resolve()
OVERWRITE = False

written, skipped, failed = [], [], []

for shp in sorted(SRC.rglob("*.shp")):
    dest = (OUT / shp.relative_to(SRC)).with_suffix(".gpkg")
    if dest.exists() and not OVERWRITE:
        skipped.append(dest)
        continue
    try:
        dest.parent.mkdir(parents=True, exist_ok=True)
        tmp = dest.with_name(dest.name + ".part")
        gpd.read_file(shp).to_file(tmp, driver="GPKG")
        tmp.replace(dest)
        written.append(dest)
    except Exception as exc:
        failed.append((shp.name, f"{type(exc).__name__}: {exc}"))

print(f"written {len(written)}, skipped {len(skipped)}, failed {len(failed)}")

Example 2: one output file, many layers 

from pathlib import Path
import geopandas as gpd

SRC = Path("data/raw").resolve()
atlas = Path("data/out/atlas.gpkg")
atlas.parent.mkdir(parents=True, exist_ok=True)
if atlas.exists():
    atlas.unlink()                     # start clean; appending to a stale file confuses runs

for shp in sorted(SRC.rglob("*.shp")):
    layer = "_".join(shp.relative_to(SRC).with_suffix("").parts).lower()
    gpd.read_file(shp).to_file(atlas, layer=layer, driver="GPKG")
    print(f"wrote layer {layer}")

Example 3: genuinely combining many inputs into one layer

If the goal was a merged dataset all along, concatenate and write once — with a column recording where each row came from.

from pathlib import Path
import geopandas as gpd
import pandas as pd

SRC = Path("data/raw").resolve()
frames = []
for shp in sorted(SRC.rglob("*.shp")):
    gdf = gpd.read_file(shp)
    gdf["source_file"] = str(shp.relative_to(SRC))
    frames.append(gdf.to_crs("EPSG:3857"))

merged = gpd.GeoDataFrame(pd.concat(frames, ignore_index=True), crs="EPSG:3857")
merged.to_file("data/out/merged.gpkg", layer="all_regions", driver="GPKG")
print(f"{len(merged)} features from {len(frames)} files")

Example 4: unique names when the stem is not enough

Some deliveries name every file export.shp inside a per-date folder. A short hash of the relative path keeps names unique and stable across runs.

import hashlib
from pathlib import Path

def unique_name(shp: Path, src_root: Path) -> str:
    rel = shp.relative_to(src_root).as_posix()
    digest = hashlib.sha1(rel.encode("utf-8")).hexdigest()[:8]
    return f"{shp.stem}_{digest}.gpkg"

A hash is stable — the same input always produces the same output name — which a counter is not.

Explanation

Writing a file is a destructive operation by default. open(path, "w"), to_file(), and every GDAL driver behind them assume that if you named a target, you meant it. There is no "are you sure?" step, because in the ordinary single-file case the overwrite is the intent.

Input directory tree mapped to an output tree preserving folder structure.
`relative_to()` carries the folder into the target path, so distinct inputs stay distinct.

A batch changes the arithmetic. With 240 inputs and one target, 239 writes are pure waste and the result is indistinguishable from a run that processed a single file. The fix is to make the mapping from input to output injective — every distinct input must produce a distinct target. The natural key is the input's path relative to the search root, because that is exactly what made the inputs distinct in the first place. Path.stem discards it; Path.relative_to() preserves it.

GeoPackage adds a second layer of naming, and understanding it removes a lot of confusion. The file is a container, and layer= names the dataset within it. Writing the same layer name repeatedly overwrites within the container; varying it accumulates. So the same to_file() call can behave like an overwrite or like an append depending on one argument — which is why a batch that "loses" its results into a single GeoPackage usually just needs a varying layer=.

The final concern is durability. A crash during a write leaves a file that exists but is not valid, and a resumable batch that checks dest.exists() will then skip it forever. Writing to name.part and renaming into place means the target only ever appears once it is complete, which makes existence a trustworthy signal of "done".

Edge cases or notes

  • Shapefile sidecars move together: Overwriting a .shp leaves stale .dbf/.shx/.prj files if the write fails partway. This is one more reason to prefer GeoPackage for outputs.
  • Case-insensitive filesystems: Roads.gpkg and roads.gpkg are the same file on Windows and default macOS. Normalise output names to lower case to avoid platform-dependent collisions.
  • Parallel workers need distinct targets: Two processes writing the same GeoPackage will corrupt it. Give each worker its own output file and merge afterwards.
  • mode="a" requires a matching schema: Appending to an existing layer fails if column names or types differ. Normalise the schema before appending.
  • Do not write into the input folder: A recursive glob will pick up your own outputs on the next run, doubling the work and eventually the data.
  • with_stem() needs Python 3.9+: On older versions use dest.with_name(f"{dest.stem}_{n}{dest.suffix}").

FAQ

Why did my batch produce only one output file?

The output path was almost certainly constant — computed above the loop, or built from a fixed string. Every iteration wrote to the same target, so only the last one survives. Build the path inside the loop from the input path.

Does to_file() overwrite a GeoPackage or add to it?

It replaces the named layer and leaves other layers alone. Vary layer= per input to accumulate layers in one file, or pass mode="a" to append rows to an existing layer.

How do I keep files with the same name from different folders apart?

Use shp.relative_to(src_root) to build the output path, which preserves the folder structure. If you need a flat output folder, join the relative parts into the filename instead: north_roads.gpkg.

Should the script overwrite existing outputs or skip them?

Skip while iterating on a long batch — it gives you a cheap resume. Overwrite for scheduled production runs that must always reflect the latest input. Whichever you choose, make it an explicit flag rather than an accident.

How do I make sure a crash does not leave a corrupt output?

Write to a temporary path such as name.gpkg.part and call Path.replace() once the write returns. The rename is atomic, so the final path only ever exists in a complete state.

Can two parallel workers write to the same GeoPackage?

No. SQLite-based formats are not safe for concurrent writers from separate processes, and you risk a corrupt file. Give each worker its own output and merge at the end.

Why do I get "No such file or directory" for a path I can see in the message?

The parent directory does not exist. Writers do not create intermediate folders, so call dest.parent.mkdir(parents=True, exist_ok=True) before every write when you are mirroring an input tree.