🔧 Fix Batch processing Batch Processing · step 22 Notebook By · Updated

Batch Script Stops at the First Bad File: How to Keep It Running

On this page

Problem statement

You point a script at a folder of 400 shapefiles, start it before lunch, and come back to a traceback from file number seven. The other 393 files were never touched. The loop did exactly what Python loops do: an unhandled exception ended it.

Traceback (most recent call last):
  File "batch_clip.py", line 18, in 
    gdf = gpd.read_file(path)
  ...
fiona.errors.DriverError: data/raw/parcels_2019.shp: No such file or directory

The failure is rarely interesting. One file is missing its .dbf sidecar, one has a null geometry, one was still being copied when the run started. What is expensive is that a single bad input costs you the entire run — and worse, you often cannot tell how far it got, because the successful outputs and the failure are interleaved in the same terminal scrollback.

Why one file takes the whole run down:

  • the whole loop body sits outside any try block, so any exception unwinds the loop
  • an error is caught, but the handler calls raise, sys.exit(), or break
  • the failure happens during file discovery rather than per file, so nothing runs at all
  • output is written to a path whose parent directory does not exist
  • the script has no record of what already succeeded, so a restart repeats everything

The fix is structural, not defensive: process each file inside its own error boundary, collect the outcome of every file, and decide at the end of the run whether the batch as a whole failed.

Quick answer

To stop one bad file from ending the run:

  1. move the per-file work into a function that takes one path and returns a result
  2. wrap the call in try / except Exception inside the loop, not around the loop
  3. record ok / failed per file with the exception message, never just print()
  4. keep going, then write a summary and a failures list at the end
  5. exit non-zero only if the failure count crosses a threshold you chose deliberately
from pathlib import Path
import geopandas as gpd

Path("data/out").mkdir(parents=True, exist_ok=True)
results = []
for path in sorted(Path("data/raw").glob("*.shp")):
    try:
        gdf = gpd.read_file(path)
        gdf.to_file(Path("data/out") / f"{path.stem}.gpkg", driver="GPKG")
        results.append({"file": path.name, "status": "ok", "error": ""})
    except Exception as exc:                      # noqa: BLE001 - batch boundary
        results.append({"file": path.name, "status": "failed",
                        "error": f"{type(exc).__name__}: {exc}"})

failed = [r for r in results if r["status"] == "failed"]
print(f"{len(results) - len(failed)} ok, {len(failed)} failed")
for r in failed:
    print(" ", r["file"], "—", r["error"][:80])
print("ocean.shp reads with columns", list(gpd.read_file("data/raw/ocean.shp").columns))
16 ok, 2 failed
  coast.shp — DataSourceError: Unable to open data/raw/coast.shx or data/raw/coast.SHX. Set SH
  land.shp — DataSourceError: 'data/raw/land.shp' not recognized as being in a supported file
ocean.shp reads with columns ['geometry']

Of the five damaged deliveries, the loop catches two — the ones GDAL cannot open at all: the shapefile with no .shx and the 404 page with a .shp name. The other three are converted and reported ok: the lakes with no CRS, the rivers with 6 of their 13 geometries missing, and the ocean with no .dbf, which reads as geometry alone with its attribute columns gone. Catching exceptions keeps the batch going; it does not tell you the output is right. That needs checks that raise — see failure policy in batch processing.

The loop now survives every file. A broad except Exception is normally a smell, but at a batch boundary it is the point: the boundary exists so that one item's problem stays that item's problem.

Where the error boundary belongs

A try around the whole loop versus a try inside the loop body. the same try, one indent aparttry around the loop✕file 1, file 2 — written✕file 3 raises✕the loop itself unwinds✕397 files never opened✕you learn about one failuretry inside the loop✓file 3 recorded as failed✓the loop keeps going✓all 400 files attempted✓one summary at the end✓you learn about every failureThe boundary belongs around one unit of work, not around the run spatialworkflow.io
The try outside the loop still loses the run; only a per-item boundary keeps it alive.

Isolate each file so one failure stays one failure

The five stages of a resilient batch: discover, isolate, record, report, exit. only the middle stage may faildiscoverfail fast if emptyprocess oneits own tryrecordok / failed + whyreportcounts + failuresexit codewhat cron readsDiscovery failures stop the run; per-file failures only stop that file spatialworkflow.io
Five stages, and only the middle one is allowed to fail.

Separate discovery from processing

Discovery failures and processing failures need different treatment. If the input folder does not exist, there is nothing to be resilient about — fail loudly and immediately. If one file inside it is broken, keep going.

from pathlib import Path
import sys

src = Path("data/raw")
if not src.is_dir():
    sys.exit(f"input folder not found: {src.resolve()}")

files = sorted(src.glob("*.shp"))
if not files:
    sys.exit(f"no .shp files matched in {src.resolve()}")

print(f"found {len(files)} files")
found 18 files

Checking up front also gives you an honest denominator. "12 of 400 failed" means something; "12 failed" on its own does not.

Put the work in a single-file function

A function that handles exactly one file is easier to test, easier to retry, and impossible to accidentally leave outside the error boundary.

import geopandas as gpd
from pathlib import Path

def process_one(path: Path, out_dir: Path) -> Path:
    """Convert one shapefile to GeoPackage. Raises on any problem."""
    gdf = gpd.read_file(path)
    if gdf.empty:
        raise ValueError("layer has no features")
    out_path = out_dir / f"{path.stem}.gpkg"
    gdf.to_file(out_path, driver="GPKG")
    return out_path

Note that the function raises. It does not print, and it does not swallow anything. Deciding what to do about a failure is the caller's job — which keeps the same function usable from a retry wrapper or a parallel worker later.

Wrap each call, and record the outcome

The loop becomes a thin driver: call, catch, record.

import traceback
from pathlib import Path

out_dir = Path("data/out")
out_dir.mkdir(parents=True, exist_ok=True)

results = []
for i, path in enumerate(files, start=1):
    print(f"[{i}/{len(files)}] {path.name}", flush=True)
    try:
        out_path = process_one(path, out_dir)
        results.append({"file": path.name, "status": "ok", "error": "", "output": str(out_path)})
    except Exception as exc:
        results.append({
            "file": path.name,
            "status": "failed",
            "error": f"{type(exc).__name__}: {exc}",
            "traceback": traceback.format_exc(),
        })
[1/18] coast.shp
[2/18] lakes.shp
[3/18] land.shp
[4/18] ne_110m_admin_0_boundary_lines_land.shp
[5/18] ne_110m_admin_0_countries.shp
[6/18] ne_110m_admin_1_states_provinces.shp
[7/18] ne_110m_coastline.shp
[8/18] ne_110m_geography_regions_polys.shp
[9/18] ne_110m_glaciated_areas.shp
[10/18] ne_110m_lakes.shp
[11/18] ne_110m_land.shp
[12/18] ne_110m_ocean.shp
[13/18] ne_110m_populated_places.shp
[14/18] ne_110m_rivers_lake_centerlines.shp
[15/18] ne_50m_admin_0_countries.shp
[16/18] ne_50m_populated_places.shp
[17/18] ocean.shp
[18/18] rivers.shp

Keeping the formatted traceback in the result — not only the message — means you can diagnose a failure days later without rerunning the batch.

Five real ways a shapefile delivery arrives damaged, and what a per-file try/except makes of each. five damaged deliveriesdeliveryreading itthe loop reportscoast.shp, no .shxDataSourceErrorfailedland.shp, a 404 pageDataSourceErrorfailedocean.shp, no .dbfreads, no attributesoklakes.shp, no .prjreads, CRS Noneokrivers.shp, cut offreads, 6 of 13 emptyokthe handler only sees what raises; three of five damages do notNatural Earth copies, damaged on purpose, converted to GeoPackage spatialworkflow.io
A per-file try keeps the batch alive; it only fails the files that raise, and three of these five do not.

Do not let the handler re-raise by accident

Three lines commonly sneak into an except block and undo the whole design:

# each of these ends the batch — usually not what you want
except Exception as exc:
    raise                      # re-raises immediately
except Exception as exc:
    sys.exit(1)                # kills the process
except Exception as exc:
    break                      # leaves the remaining files unprocessed

Use continue (or simply fall through to the next iteration). break belongs only in a deliberate "stop after N consecutive failures" circuit breaker.

Catch the exceptions that do not subclass Exception

except Exception does not catch KeyboardInterrupt or SystemExit, and that is correct — Ctrl-C should still stop the run. But if a C library segfaults or a worker is killed by the OOM killer, no Python handler runs at all. Those failures need process-level protection, which is why long batches write their progress to disk as they go rather than only at the end.

import csv

with open("run_log.csv", "w", newline="", encoding="utf-8") as fh:
    writer = csv.DictWriter(fh, fieldnames=["file", "status", "error"])
    writer.writeheader()
    for path in files:
        row = {"file": path.name, "status": "ok", "error": ""}
        try:
            process_one(path, out_dir)
        except Exception as exc:
            row = {"file": path.name, "status": "failed", "error": f"{type(exc).__name__}: {exc}"}
        writer.writerow(row)
        fh.flush()          # survive a hard kill

Report once, at the end

A batch that "keeps going" without a summary just moves the problem: now the failures are buried in 400 lines of output.

failed = [r for r in results if r["status"] == "failed"]

print("\n─── batch summary ───")
print(f"total   : {len(results)}")
print(f"ok      : {len(results) - len(failed)}")
print(f"failed  : {len(failed)}")
for r in failed:
    print(f"  ! {r['file']}: {r['error'][:70]}")

exit_code = 1 if len(failed) > len(results) * 0.1 else 0     # your threshold, chosen deliberately
print("exit code:", exit_code)                               # a script ends with sys.exit(exit_code)

─── batch summary ───
total   : 18
ok      : 16
failed  : 2
  ! coast.shp: DataSourceError: Unable to open data/raw/coast.shx or data/raw/coast.S
  ! land.shp: DataSourceError: 'data/raw/land.shp' not recognized as being in a supp
exit code: 1

The exit code matters more than it looks: a scheduler, a CI job, or a Makefile decides whether the run "worked" from that number alone.

A resilient converter for a damaged folder

The data used on this page

A real folder of shapefiles: thirteen layers of Natural Earth (public domain) at 1:110 million and 1:50 million, flat in data/raw/. Five more are copies damaged on purpose, each in a way deliveries really arrive: lakes.shp without its .prj; rivers.shp cut off halfway, as an interrupted copy leaves it; coast.shp without its .shx; ocean.shp without its .dbf; and land.shp, which is a web server's 404 page saved under a shapefile's name.

import io
import shutil
import zipfile
from pathlib import Path

import requests

NE = {"cultural": ["ne_110m_admin_0_countries", "ne_110m_admin_1_states_provinces", "ne_110m_populated_places",
                   "ne_110m_admin_0_boundary_lines_land", "ne_50m_admin_0_countries", "ne_50m_populated_places"],
      "physical": ["ne_110m_coastline", "ne_110m_land", "ne_110m_ocean", "ne_110m_rivers_lake_centerlines",
                   "ne_110m_lakes", "ne_110m_glaciated_areas", "ne_110m_geography_regions_polys"]}
SRC = Path("data/ne")
for d in ("data/raw", "data/out"):
    shutil.rmtree(d, ignore_errors=True)               # every run of the page starts from nothing
for theme, names in NE.items():
    for name in names:
        if not (SRC / theme / f"{name}.shp").exists():
            scale = name.split("_")[1]
            reply = requests.get(f"https://naciscdn.org/naturalearth/{scale}/{theme}/{name}.zip", timeout=120)
            reply.raise_for_status()
            zipfile.ZipFile(io.BytesIO(reply.content)).extractall(SRC / theme)

raw = Path("data/raw")
raw.mkdir(parents=True)
for f in SRC.rglob("*"):
    if f.suffix in (".shp", ".shx", ".dbf", ".prj", ".cpg"):
        shutil.copy(f, raw / f.name)

# five deliveries damaged on purpose, each the way files really arrive damaged
def deliver(layer, name, parts):
    for ext in parts:
        shutil.copy(SRC / "physical" / f"ne_110m_{layer}.{ext}", raw / f"{name}.{ext}")

deliver("lakes", "lakes", ["shp", "shx", "dbf"])                      # no .prj
deliver("rivers_lake_centerlines", "rivers", ["shp", "shx", "dbf", "prj"])
data = (raw / "rivers.shp").read_bytes()
(raw / "rivers.shp").write_bytes(data[: len(data) // 2])              # cut off mid-copy
deliver("coastline", "coast", ["shp", "dbf", "prj"])                  # no .shx
deliver("ocean", "ocean", ["shp", "shx", "prj"])                      # no .dbf
deliver("land", "land", ["shx", "dbf", "prj"])
(raw / "land.shp").write_text("<html><body><h1>404 Not Found</h1></body></html>")

print(len(sorted(raw.glob("*.shp"))), "shapefiles in", raw)
18 shapefiles in data/raw

Example 1: a complete resilient batch converter

from pathlib import Path
import sys
import traceback
import geopandas as gpd

SRC = Path("data/raw")
OUT = Path("data/out")

def process_one(path: Path) -> Path:
    gdf = gpd.read_file(path)
    if gdf.empty:
        raise ValueError("layer has no features")
    out_path = OUT / f"{path.stem}.gpkg"
    gdf.to_file(out_path, driver="GPKG")
    return out_path

def main() -> int:
    if not SRC.is_dir():
        print(f"input folder not found: {SRC.resolve()}", file=sys.stderr)
        return 2
    files = sorted(SRC.glob("*.shp"))
    if not files:
        print(f"nothing to do in {SRC.resolve()}", file=sys.stderr)
        return 2

    OUT.mkdir(parents=True, exist_ok=True)
    ok, failed = [], []

    for i, path in enumerate(files, start=1):
        print(f"[{i}/{len(files)}] {path.name}", flush=True)
        try:
            ok.append(process_one(path))
        except Exception as exc:
            failed.append((path.name, f"{type(exc).__name__}: {exc}"))
            print(f"    failed: {exc}", file=sys.stderr, flush=True)

    print(f"\n{len(ok)} ok, {len(failed)} failed")
    for name, err in failed:
        print(f"  ! {name}: {err}")
    return 1 if failed else 0

if __name__ == "__main__":
    raise SystemExit(main())

Run as a script, that is the whole file. Here, call it:

code = main()
print("exit code:", code)
[1/18] coast.shp
[2/18] lakes.shp
[3/18] land.shp
[4/18] ne_110m_admin_0_boundary_lines_land.shp
[5/18] ne_110m_admin_0_countries.shp
[6/18] ne_110m_admin_1_states_provinces.shp
[7/18] ne_110m_coastline.shp
[8/18] ne_110m_geography_regions_polys.shp
[9/18] ne_110m_glaciated_areas.shp
[10/18] ne_110m_lakes.shp
[11/18] ne_110m_land.shp
[12/18] ne_110m_ocean.shp
[13/18] ne_110m_populated_places.shp
[14/18] ne_110m_rivers_lake_centerlines.shp
[15/18] ne_50m_admin_0_countries.shp
[16/18] ne_50m_populated_places.shp
[17/18] ocean.shp
[18/18] rivers.shp

16 ok, 2 failed
  ! coast.shp: DataSourceError: Unable to open data/raw/coast.shx or data/raw/coast.SHX. Set SHAPE_RESTORE_SHX config option to YES to restore or create it.
  ! land.shp: DataSourceError: 'data/raw/land.shp' not recognized as being in a supported file format.; It might help to specify the correct driver explicitly by prefixing the file path with '<DRIVER>:', e.g. 'CSV:path'.
exit code: 1

Example 2: stop the run only after repeated failures

If every file is failing, continuing is a waste of an hour. A consecutive-failure circuit breaker gets both behaviours.

consecutive = 0
LIMIT = 5

for path in files:
    try:
        process_one(path)
        consecutive = 0
    except Exception as exc:
        consecutive += 1
        print(f"failed {path.name}: {exc}", file=sys.stderr)
        if consecutive >= LIMIT:
            print(f"aborting: {LIMIT} failures in a row", file=sys.stderr)
            break

A shared cause — an unmounted drive, an expired database password — shows up as a run of failures, not as scattered ones.

Example 3: separate expected failures from bugs

Not every exception deserves the same reaction. A missing sidecar file is data; a TypeError in your own code is a bug you want to see.

from pyogrio.errors import DataSourceError     # what GeoPandas raises now; fiona's DriverError no longer reaches you

failed = []
for path in files:
    try:
        process_one(path)
    except (DataSourceError, ValueError) as exc:
        # data problems: record and move on
        failed.append((path.name, str(exc)))
    except Exception:
        # programming errors: stop, because every file will hit them
        print(f"unexpected error on {path.name}", file=sys.stderr)
        raise
print([name for name, _ in failed])
['coast.shp', 'land.shp']

The first version of this block caught fiona.errors.DriverError. GeoPandas reads through pyogrio by default now, which raises its own DataSourceError, so the missing .shx fell through to the "programming error" branch and stopped the run — the exact failure the page is about, caused by the handler meant to prevent it.

Example 4: a failures manifest you can re-run

import json
import shutil
from pathlib import Path
from pyogrio import set_gdal_config_options

Path("failures.json").write_text(
    json.dumps([{"file": n, "error": e} for n, e in failed], indent=2),
    encoding="utf-8",
)

# later, after fixing the inputs: rebuild coast's .shx from the .shp, and fetch land again
set_gdal_config_options({"SHAPE_RESTORE_SHX": "YES"})
gpd.read_file(SRC / "coast.shp")                            # writes coast.shx as a side effect
set_gdal_config_options({"SHAPE_RESTORE_SHX": None})
shutil.copy("data/ne/physical/ne_110m_land.shp", SRC / "land.shp")

retry = [SRC / item["file"] for item in json.loads(Path("failures.json").read_text())]
for path in retry:
    print(path.name, "→", process_one(path).name)
coast.shp → coast.gpkg
land.shp → land.gpkg

Re-running only the failures turns a two-hour batch into a two-minute one.

Why an exception escapes the loop

An unhandled exception in Python unwinds the call stack until something catches it. A for loop is not a barrier — it is part of that stack. So an exception raised on iteration seven propagates straight out of the loop, past any code that came after it, and terminates the program. Nothing about the loop "remembers" that there were 393 iterations left.

Six ways a batch GIS job dies, and the correct response to each. six ways a batch diesWhat happenedWhat to doMissing .dbf or .shx sidecarRecord it, skip, list in thereportNull or invalid geometryRepair in the step, or skip therowOutput folder does not existmkdir(parents=True) before theloopHandler calls raise or breakUse continue; break only on astreakWorker killed by the OOM killerFlush a progress log after eachfileDisk full or permission deniedCircuit-break after N failures ina rowTwo of these should stop the run — the other four should not spatialworkflow.io
Most batch failures are one of five things, and each has a different correct response.

Wrapping the loop in try does not help, because the exception still escapes the loop before the handler runs — you catch it, but the remaining iterations are gone. The boundary has to be inside the loop body, around one unit of work. That is the whole idea: define the unit of work, give it its own error boundary, and let the driver decide what a failure means.

Once each item is isolated, the interesting question changes from "did it crash?" to "what happened to each item?" That is why the result list matters as much as the try. A batch run has three possible outcomes — everything worked, some things worked, nothing worked — and only a per-item record can distinguish the middle case, which is by far the most common one in real GIS data.

The final piece is the exit code. Inside an automated pipeline, nobody reads your summary; a scheduler reads $?. Choosing "non-zero if any file failed" or "non-zero if more than 10% failed" is a policy decision, and making it explicitly is what separates a script that runs unattended from one that merely runs.

What except Exception still misses

  • except Exception will not catch everything: KeyboardInterrupt and SystemExit derive from BaseException, so Ctrl-C still stops the run — which is what you want. Never use bare except: to "fix" that.
  • Segfaults skip Python entirely: A crash inside GDAL or a killed process leaves no traceback. Flush a progress log after each file so a hard kill still tells you where it stopped.
  • Partial output files are still files: If to_file() fails halfway, a truncated output may exist. Write to a temporary name and rename on success so downstream steps never see a half-written file.
  • Logging beats printing for long runs: logging gives you timestamps, levels, and a file handler for free, and it is thread-safe under parallel execution.
  • Do not catch inside process_one: If the worker function swallows its own errors and returns None, the driver cannot tell success from failure. Let it raise.
  • Order affects debuggability: sorted() on the file list makes a run reproducible, so "it failed on the seventh file" means the same thing tomorrow.

FAQ

Is a broad except Exception not bad practice?

It is, everywhere except at a boundary where you have deliberately decided that one item's failure must not affect the others. The rule that makes it safe is that you record the exception rather than discard it — a broad catch with a full traceback in the log is very different from except: pass.

Should the script exit 0 or 1 when some files failed?

Exit non-zero if a human needs to look at the run. For most GIS batches that means non-zero when any file failed, because a silently skipped file becomes missing data downstream. Loosen it to a percentage threshold only when partial failure is genuinely acceptable.

How do I re-run only the files that failed?

Write the failures to a JSON or CSV manifest during the run, then feed that manifest back in as the input list. This is also the foundation of a resumable job, where the manifest of completed work lives on disk between runs.

Why does my try/except still stop the loop?

Check whether the handler calls raise, break, or sys.exit(), and whether the try is actually inside the loop body. A try wrapped around the entire for statement catches the exception but the loop has already ended by then.

Can I keep the traceback without printing it for every file?

Yes — store traceback.format_exc() in the result dictionary and print only the one-line message during the run. Dump the full tracebacks to a log file at the end, or on demand with a --verbose flag.

Does this work the same with a ThreadPoolExecutor or ProcessPoolExecutor?

Yes, and it is cleaner there: an exception raised in a worker is stored in its Future and re-raised when you call future.result(). Wrap that call in try/except and you get the same per-item boundary without changing process_one.

What if the failure is in writing the output, not reading the input?

Treat it the same way, but make sure the output directory exists before the loop starts and write via a temporary file. A permissions or disk-full error will affect every file, so a consecutive-failure circuit breaker stops the run quickly instead of grinding through 400 identical errors.