Batch Output Files Keep Overwriting Each Other: How to Fix It

On this page

Problem statement

The batch run reports 13 files processed. The output folder contains one.

processed 13 files
$ ls data/out
output.gpkg

Or the count is right but the contents are wrong: roads.gpkg in the output holds the last region's roads, and 11 earlier regions have vanished. Nothing raised an error — every write succeeded. They just all wrote to the same place.

Where this comes from:

the output filename is a constant, built outside the loop
a recursive input walk flattens the tree, so north/roads.shp and south/roads.shp both become roads.gpkg
to_file() on an existing GeoPackage replaces the layer instead of adding one
the output name is derived from a column value that repeats
a parallel run has two workers writing the same path at the same time
the output directory is also the input directory, so the second pass consumes its own results

The symptom is silent because writing a file that already exists is a normal operation. Nothing in GeoPandas, Fiona, or the filesystem considers it an error, so the loss is only visible when you count the results.

Quick answer

To stop outputs from colliding:

derive the output name from the input path inside the loop — never a fixed string
preserve the input folder structure with relative_to() instead of flattening it
add a discriminator (region, date, layer name) when stems can repeat
check for an existing target before writing, and either skip, version, or fail
write to a temporary name and rename on success so partial files are never mistaken for results

from pathlib import Path
import geopandas as gpd

SRC, OUT = Path("data/raw").resolve(), Path("data/out").resolve()

for shp in sorted(SRC.rglob("*.shp")):
    rel = shp.relative_to(SRC).with_suffix(".gpkg")   # 50m/cultural/admin_0_countries.gpkg
    dest = OUT / rel
    dest.parent.mkdir(parents=True, exist_ok=True)
    if dest.exists():
        raise FileExistsError(f"refusing to overwrite {dest}")
    gpd.read_file(shp).to_file(dest, driver="GPKG")

print(len(list(OUT.rglob("*.gpkg"))), "outputs for", len(list(SRC.rglob("*.shp"))), "inputs")

13 outputs for 13 inputs

The two lines that matter are relative_to() — which keeps north/ and south/ apart — and the exists() guard, which converts a silent overwrite into a loud failure.

How outputs collide

Flattening a tree collapses distinct inputs onto one name — the last writer wins.

Compute each output path inside the loop

Four ways to build a unique output path, and what each one is good for.

Move the output path inside the loop

The classic bug is a path computed once, above the loop, and reused for every iteration.

import shutil
files = sorted(SRC.rglob("*.shp"))
shutil.rmtree("data/out", ignore_errors=True)
Path("data/out").mkdir(parents=True)

# wrong: one target for every input
out_path = Path("data/out/output.gpkg")
for shp in files:
    gpd.read_file(shp).to_file(out_path, driver="GPKG")
print(f"processed {len(files)} files ->", sorted(p.name for p in Path("data/out").iterdir()))

# right: one target per input
for shp in files:
    out_path = Path("data/out") / f"{shp.stem}.gpkg"
    gpd.read_file(shp).to_file(out_path, driver="GPKG")
print(f"one per input: {len(list(Path('data/out').glob('*.gpkg'))) - 1} files for {len(files)} inputs")

processed 13 files -> ['output.gpkg']
one per input: 11 files for 13 inputs

The first loop leaves one file — the last input's — and the second is only right for a flat folder. In this library admin_0_countries.shp and populated_places.shp exist at both scales, so naming by stem writes eleven files for thirteen inputs, and the 1:50m layers silently replace the 1:110m ones. The next step keeps the folder structure.

If you want everything in one file, that is a different operation — concatenate the frames and write once, or append layers explicitly (see below). Repeatedly overwriting is never the way to combine data.

Mirror the input tree instead of flattening it

Path.stem throws away the folder, which is exactly the information that made two same-named files distinct.

from pathlib import Path

SRC = Path("data/raw").resolve()
OUT = Path("data/out").resolve()

for shp in sorted(SRC.rglob("*.shp")):
    rel = shp.relative_to(SRC)                 # north/roads.shp
    dest = (OUT / rel).with_suffix(".gpkg")    # data/out/north/roads.gpkg
    dest.parent.mkdir(parents=True, exist_ok=True)

mkdir(parents=True, exist_ok=True) is required — writers do not create missing intermediate directories, and the resulting error is a confusing "No such file or directory" on a path you can see in the message.

Or flatten deliberately, with the folder in the name

Sometimes a flat output folder is genuinely easier to consume. Then encode the structure in the filename rather than discarding it.

rel = shp.relative_to(SRC).with_suffix("")     # north/roads
flat_name = "_".join(rel.parts) + ".gpkg"      # north_roads.gpkg
dest = OUT / flat_name

This keeps a single directory listing while guaranteeing uniqueness, and the origin of each file remains readable.

Naming by stem looks right until two folders hold a file of the same name; here it silently replaced two of thirteen.

Detect collisions before writing anything

The cheapest fix is to compute every output path up front and check for duplicates. A collision found in the first second is far better than one found after two hours.

from collections import Counter

def collisions(naming):
    targets = {}
    for shp in sorted(SRC.rglob("*.shp")):
        targets.setdefault(naming(shp), []).append(shp)
    return {d: srcs for d, srcs in targets.items() if len(srcs) > 1}

for label, naming in (("by stem", lambda shp: OUT / f"{shp.stem}.gpkg"),
                      ("by relative path", lambda shp: (OUT / shp.relative_to(SRC)).with_suffix(".gpkg"))):
    clashes = collisions(naming)
    print(f"{label}: {len(clashes)} collisions")
    for dest, srcs in clashes.items():
        print(f"  ! {dest.name} would be written by {len(srcs)} inputs:",
              ", ".join(str(s.relative_to(SRC)) for s in srcs))
# in a script: if clashes: raise SystemExit("output paths collide — fix the naming rule")

by stem: 2 collisions
  ! admin_0_countries.gpkg would be written by 2 inputs: 110m/cultural/admin_0_countries.shp, 50m/cultural/admin_0_countries.shp
  ! populated_places.gpkg would be written by 2 inputs: 110m/cultural/populated_places.shp, 50m/cultural/populated_places.shp
by relative path: 0 collisions

Understand what `to_file()` does to an existing GeoPackage

This one surprises people, because it is not a filesystem overwrite. Writing a GeoPackage that already exists replaces the layer of that name and leaves other layers intact — unless you ask for something else.

import geopandas as gpd

gdf = gpd.read_file(SRC / "110m" / "physical" / "lakes.shp")

# replaces the layer "lakes" inside atlas.gpkg, keeps other layers
gdf.to_file("atlas.gpkg", layer="lakes", driver="GPKG")
gdf.to_file("atlas.gpkg", layer="lakes", driver="GPKG")
print("after writing twice:", len(gpd.read_file("atlas.gpkg", layer="lakes")), "rows")

# adds rows to an existing layer instead of replacing it
gdf.to_file("atlas.gpkg", layer="lakes", driver="GPKG", mode="a")
print("after mode='a':     ", len(gpd.read_file("atlas.gpkg", layer="lakes")), "rows")

after writing twice: 24 rows
after mode='a':      48 rows

So a loop that writes every region to atlas.gpkg with the same layer= name ends up with only the last region — but a loop that varies layer= produces a single, tidy multi-layer file. That is often exactly what you want.

for shp in sorted(SRC.rglob("*.shp")):
    layer = "_".join(shp.relative_to(SRC).with_suffix("").parts)
    gpd.read_file(shp).to_file(OUT / "atlas.gpkg", layer=layer, driver="GPKG")

Shapefiles have no such concept: one shapefile is one layer, so unique filenames are the only option there.

Write atomically so failures cannot corrupt a good file

If the process dies mid-write, you are left with a file that exists but is incomplete — and the next run will happily treat it as done.

from pathlib import Path
import geopandas as gpd

def write_atomic(gdf, dest: Path, driver="GPKG"):
    dest.parent.mkdir(parents=True, exist_ok=True)
    tmp = dest.with_suffix(".part" + dest.suffix)   # roads.part.gpkg: GDAL warns on any other extension
    gdf.to_file(tmp, driver=driver)
    tmp.replace(dest)          # atomic on the same filesystem

Path.replace() is an atomic rename on POSIX and overwrites on Windows too, so a reader never observes a half-written result.

Decide the overwrite policy explicitly

Every batch needs one of three behaviours, chosen on purpose:

MODE = "skip"      # "skip" | "overwrite" | "version" | "fail"

if dest.exists():
    if MODE == "skip":
        continue
    if MODE == "fail":
        raise FileExistsError(dest)
    if MODE == "version":
        n = 1
        while dest.with_stem(f"{dest.stem}_{n}").exists():
            n += 1
        dest = dest.with_stem(f"{dest.stem}_{n}")
    # "overwrite" falls through and writes

skip doubles as a crude resume mechanism; fail is the right default while you are still developing the naming rule.

Safe outputs for same-named files in different folders

The data used on this page

A real nested library of shapefiles: thirteen layers of Natural Earth (public domain), filed by scale and theme as data/raw/<scale>/<theme>/<layer>.shp — so 110m/cultural/admin_0_countries.shp and 50m/cultural/admin_0_countries.shp are different files with the same name. Thirteen inputs, eleven distinct file names.

import io
import shutil
import zipfile
from pathlib import Path

import geopandas as gpd
import requests

NE = {"cultural": ["ne_110m_admin_0_countries", "ne_110m_admin_1_states_provinces", "ne_110m_populated_places",
                   "ne_110m_admin_0_boundary_lines_land", "ne_50m_admin_0_countries", "ne_50m_populated_places"],
      "physical": ["ne_110m_coastline", "ne_110m_land", "ne_110m_ocean", "ne_110m_rivers_lake_centerlines",
                   "ne_110m_lakes", "ne_110m_glaciated_areas", "ne_110m_geography_regions_polys"]}
raw = Path("data/raw")
for d in ("data/out",):
    shutil.rmtree(d, ignore_errors=True)              # every run of the page starts from nothing
for theme, names in NE.items():
    for name in names:
        scale, layer = name.split("_")[1], name.split("_", 2)[2]
        folder = raw / scale / theme
        if not (folder / f"{layer}.shp").exists():
            reply = requests.get(f"https://naciscdn.org/naturalearth/{scale}/{theme}/{name}.zip", timeout=120)
            reply.raise_for_status()
            with zipfile.ZipFile(io.BytesIO(reply.content)) as z:
                for member in z.namelist():
                    if member.startswith(name + "."):
                        folder.mkdir(parents=True, exist_ok=True)
                        (folder / member.replace(name, layer)).write_bytes(z.read(member))
print(len(sorted(raw.rglob("*.shp"))), "shapefiles in", sorted({str(p.parent.relative_to(raw)) for p in raw.rglob("*.shp")}))

13 shapefiles in ['110m/cultural', '110m/physical', '50m/cultural']

Example 1: safe per-file conversion with a mirrored tree

from pathlib import Path
import geopandas as gpd

SRC = Path("data/raw").resolve()
OUT = Path("data/out").resolve()
OVERWRITE = False

def run():
    written, skipped, failed = [], [], []
    for shp in sorted(SRC.rglob("*.shp")):
        dest = (OUT / shp.relative_to(SRC)).with_suffix(".gpkg")
        if dest.exists() and not OVERWRITE:
            skipped.append(dest)
            continue
        try:
            dest.parent.mkdir(parents=True, exist_ok=True)
            tmp = dest.with_suffix(".part.gpkg")
            gpd.read_file(shp).to_file(tmp, driver="GPKG")
            tmp.replace(dest)
            written.append(dest)
        except Exception as exc:
            failed.append((shp.name, f"{type(exc).__name__}: {exc}"))
    print(f"written {len(written)}, skipped {len(skipped)}, failed {len(failed)}")

shutil.rmtree(OUT, ignore_errors=True)
run()
run()          # a re-run skips what is there

written 13, skipped 0, failed 0
written 0, skipped 13, failed 0

Example 2: one output file, many layers

from pathlib import Path
import geopandas as gpd
import pyogrio

SRC = Path("data/raw").resolve()
atlas = Path("data/out/atlas.gpkg")
atlas.parent.mkdir(parents=True, exist_ok=True)
if atlas.exists():
    atlas.unlink()                     # start clean; appending to a stale file confuses runs

for shp in sorted(SRC.rglob("*.shp")):
    layer = "_".join(shp.relative_to(SRC).with_suffix("").parts).lower()
    gpd.read_file(shp).to_file(atlas, layer=layer, driver="GPKG")
print(len(pyogrio.list_layers(atlas)), "layers, e.g.", [name for name, _ in pyogrio.list_layers(atlas)[-2:]])

13 layers, e.g. ['50m_cultural_admin_0_countries', '50m_cultural_populated_places']

Example 3: genuinely combining many inputs into one layer

If the goal was a merged dataset all along, concatenate and write once — with a column recording where each row came from.

from pathlib import Path
import geopandas as gpd
import pandas as pd

SRC = Path("data/raw").resolve()
frames = []
for shp in sorted(SRC.rglob("*.shp")):
    gdf = gpd.read_file(shp)
    gdf = gdf.rename(columns=str.lower)        # GeoPackage field names are case-insensitive
    gdf["source_file"] = str(shp.relative_to(SRC))
    frames.append(gdf.to_crs("EPSG:8857"))

merged = gpd.GeoDataFrame(pd.concat(frames, ignore_index=True), crs="EPSG:8857")
merged.to_file("data/out/merged.gpkg", layer="all_regions", driver="GPKG")
print(f"{len(merged)} features from {len(frames)} files, {merged.geom_type.nunique()} geometry types, "
      f"{merged.shape[1] - 2} attribute columns")
print(pyogrio.read_info("data/out/merged.gpkg", layer="all_regions")["geometry_type"])

2666 features from 13 files, 5 geometry types, 274 attribute columns
Unknown

The rename is there because the first run of this block failed with FieldError: Error adding field 'featurecla'. Natural Earth is not consistent about case — 53 column names appear as both name and NAME, featurecla and FEATURECLA across these layers — pandas treats them as different columns, and GeoPackage, whose field names are case-insensitive, will not create the second. Merging many sources into one layer also produces one very wide table, and a layer with five geometry types, which GDAL can only record as Unknown and some readers refuse; separate layers in one file (Example 2) avoid both.

Example 4: unique names when the stem is not enough

Some deliveries name every file export.shp inside a per-date folder. A short hash of the relative path keeps names unique and stable across runs.

import hashlib
from pathlib import Path

def unique_name(shp: Path, src_root: Path) -> str:
    rel = shp.relative_to(src_root).as_posix()
    digest = hashlib.sha1(rel.encode("utf-8")).hexdigest()[:8]
    return f"{shp.stem}_{digest}.gpkg"

for shp in sorted(SRC.rglob("admin_0_countries.shp")):
    print(shp.relative_to(SRC), "->", unique_name(shp, SRC))

110m/cultural/admin_0_countries.shp -> admin_0_countries_f3ba0f4e.gpkg
50m/cultural/admin_0_countries.shp -> admin_0_countries_39b5416c.gpkg

A hash is stable — the same input always produces the same output name — which a counter is not.

Why writing a file destroys the old one

Writing a file is a destructive operation by default. open(path, "w"), to_file(), and every GDAL driver behind them assume that if you named a target, you meant it. There is no "are you sure?" step, because in the ordinary single-file case the overwrite is the intent.

relative_to() carries the folder into the target path, so distinct inputs stay distinct.

A batch changes the arithmetic. With 13 inputs and one target, 12 writes are pure waste and the result is indistinguishable from a run that processed a single file. The fix is to make the mapping from input to output injective — every distinct input must produce a distinct target. The natural key is the input's path relative to the search root, because that is exactly what made the inputs distinct in the first place. Path.stem discards it; Path.relative_to() preserves it.

GeoPackage adds a second layer of naming, and understanding it removes a lot of confusion. The file is a container, and layer= names the dataset within it. Writing the same layer name repeatedly overwrites within the container; varying it accumulates. So the same to_file() call can behave like an overwrite or like an append depending on one argument — which is why a batch that "loses" its results into a single GeoPackage usually just needs a varying layer=.

The final concern is durability. A crash during a write leaves a file that exists but is not valid, and a resumable batch that checks dest.exists() will then skip it forever. Writing to name.part and renaming into place means the target only ever appears once it is complete, which makes existence a trustworthy signal of "done".

Sidecars and case-insensitive filesystems

Shapefile sidecars move together: Overwriting a .shp leaves stale .dbf/.shx/.prj files if the write fails partway. This is one more reason to prefer GeoPackage for outputs.
Case-insensitive filesystems: Roads.gpkg and roads.gpkg are the same file on Windows and default macOS. Normalise output names to lower case to avoid platform-dependent collisions.
Parallel workers need distinct targets: Two processes writing the same GeoPackage will corrupt it. Give each worker its own output file and merge afterwards.
mode="a" requires a matching schema: Appending to an existing layer fails if column names or types differ. Normalise the schema before appending.
Do not write into the input folder: A recursive glob will pick up your own outputs on the next run, doubling the work and eventually the data.
with_stem() needs Python 3.9+: On older versions use dest.with_name(f"{dest.stem}_{n}{dest.suffix}").

Internal links

FAQ

Why did my batch produce only one output file?

The output path was almost certainly constant — computed above the loop, or built from a fixed string. Every iteration wrote to the same target, so only the last one survives. Build the path inside the loop from the input path.

Does `to_file()` overwrite a GeoPackage or add to it?

It replaces the named layer and leaves other layers alone. Vary layer= per input to accumulate layers in one file, or pass mode="a" to append rows to an existing layer.

How do I keep files with the same name from different folders apart?

Use shp.relative_to(src_root) to build the output path, which preserves the folder structure. If you need a flat output folder, join the relative parts into the filename instead: north_roads.gpkg.

Should the script overwrite existing outputs or skip them?

Skip while iterating on a long batch — it gives you a cheap resume. Overwrite for scheduled production runs that must always reflect the latest input. Whichever you choose, make it an explicit flag rather than an accident.

How do I make sure a crash does not leave a corrupt output?

Write to a temporary path such as name.gpkg.part and call Path.replace() once the write returns. The rename is atomic, so the final path only ever exists in a complete state.

Can two parallel workers write to the same GeoPackage?

No. SQLite-based formats are not safe for concurrent writers from separate processes, and you risk a corrupt file. Give each worker its own output and merge at the end.

Why do I get "No such file or directory" for a path I can see in the message?

The parent directory does not exist. Writers do not create intermediate folders, so call dest.parent.mkdir(parents=True, exist_ok=True) before every write when you are mirroring an input tree.