🔧 Fix Batch processing Batch Processing · step 22 of 26 Updated

Batch Script Stops at the First Bad File: How to Keep It Running

Problem statement

You point a script at a folder of 400 shapefiles, start it before lunch, and come back to a traceback from file number seven. The other 393 files were never touched. The loop did exactly what Python loops do: an unhandled exception ended it.

Traceback (most recent call last):
  File "batch_clip.py", line 18, in 
    gdf = gpd.read_file(path)
  ...
fiona.errors.DriverError: data/raw/parcels_2019.shp: No such file or directory

The failure is rarely interesting. One file is missing its .dbf sidecar, one has a null geometry, one was still being copied when the run started. What is expensive is that a single bad input costs you the entire run — and worse, you often cannot tell how far it got, because the successful outputs and the failure are interleaved in the same terminal scrollback.

Common causes:

  • the whole loop body sits outside any try block, so any exception unwinds the loop
  • an error is caught, but the handler calls raise, sys.exit(), or break
  • the failure happens during file discovery rather than per file, so nothing runs at all
  • output is written to a path whose parent directory does not exist
  • the script has no record of what already succeeded, so a restart repeats everything

The fix is structural, not defensive: process each file inside its own error boundary, collect the outcome of every file, and decide at the end of the run whether the batch as a whole failed.

Quick answer

To stop one bad file from ending the run:

  1. move the per-file work into a function that takes one path and returns a result
  2. wrap the call in try / except Exception inside the loop, not around the loop
  3. record ok / failed per file with the exception message, never just print()
  4. keep going, then write a summary and a failures list at the end
  5. exit non-zero only if the failure count crosses a threshold you chose deliberately
from pathlib import Path
import geopandas as gpd

results = []
for path in sorted(Path("data/raw").glob("*.shp")):
    try:
        gdf = gpd.read_file(path)
        gdf.to_file(Path("data/out") / f"{path.stem}.gpkg", driver="GPKG")
        results.append({"file": path.name, "status": "ok", "error": ""})
    except Exception as exc:                      # noqa: BLE001 - batch boundary
        results.append({"file": path.name, "status": "failed",
                        "error": f"{type(exc).__name__}: {exc}"})

failed = [r for r in results if r["status"] == "failed"]
print(f"{len(results) - len(failed)} ok, {len(failed)} failed")

The loop now survives every file. A broad except Exception is normally a smell, but at a batch boundary it is the point: the boundary exists so that one item's problem stays that item's problem.

Where the error boundary belongs

Two loops compared — a try around the whole loop stops at file three, a try inside the loop processes all files.
The `try` outside the loop still loses the run; only a per-item boundary keeps it alive.

Step-by-step solution

Per-file pipeline — discover, isolate, record, report, exit code.
Five stages, and only the middle one is allowed to fail.

Separate discovery from processing

Discovery failures and processing failures need different treatment. If the input folder does not exist, there is nothing to be resilient about — fail loudly and immediately. If one file inside it is broken, keep going.

from pathlib import Path
import sys

src = Path("data/raw")
if not src.is_dir():
    sys.exit(f"input folder not found: {src.resolve()}")

files = sorted(src.glob("*.shp"))
if not files:
    sys.exit(f"no .shp files matched in {src.resolve()}")

print(f"found {len(files)} files")

Checking up front also gives you an honest denominator. "12 of 400 failed" means something; "12 failed" on its own does not.

Put the work in a single-file function

A function that handles exactly one file is easier to test, easier to retry, and impossible to accidentally leave outside the error boundary.

import geopandas as gpd
from pathlib import Path

def process_one(path: Path, out_dir: Path) -> Path:
    """Convert one shapefile to GeoPackage. Raises on any problem."""
    gdf = gpd.read_file(path)
    if gdf.empty:
        raise ValueError("layer has no features")
    out_path = out_dir / f"{path.stem}.gpkg"
    gdf.to_file(out_path, driver="GPKG")
    return out_path

Note that the function raises. It does not print, and it does not swallow anything. Deciding what to do about a failure is the caller's job — which keeps the same function usable from a retry wrapper or a parallel worker later.

Wrap each call, and record the outcome

The loop becomes a thin driver: call, catch, record.

import traceback
from pathlib import Path

out_dir = Path("data/out")
out_dir.mkdir(parents=True, exist_ok=True)

results = []
for i, path in enumerate(files, start=1):
    print(f"[{i}/{len(files)}] {path.name}", flush=True)
    try:
        out_path = process_one(path, out_dir)
        results.append({"file": path.name, "status": "ok", "error": "", "output": str(out_path)})
    except Exception as exc:
        results.append({
            "file": path.name,
            "status": "failed",
            "error": f"{type(exc).__name__}: {exc}",
            "traceback": traceback.format_exc(),
        })

Keeping the formatted traceback in the result — not only the message — means you can diagnose a failure days later without rerunning the batch.

Do not let the handler re-raise by accident

Three lines commonly sneak into an except block and undo the whole design:

# each of these ends the batch — usually not what you want
except Exception as exc:
    raise                      # re-raises immediately
except Exception as exc:
    sys.exit(1)                # kills the process
except Exception as exc:
    break                      # leaves the remaining files unprocessed

Use continue (or simply fall through to the next iteration). break belongs only in a deliberate "stop after N consecutive failures" circuit breaker.

Catch the exceptions that do not subclass Exception

except Exception does not catch KeyboardInterrupt or SystemExit, and that is correct — Ctrl-C should still stop the run. But if a C library segfaults or a worker is killed by the OOM killer, no Python handler runs at all. Those failures need process-level protection, which is why long batches write their progress to disk as they go rather than only at the end.

import csv

with open("run_log.csv", "w", newline="", encoding="utf-8") as fh:
    writer = csv.DictWriter(fh, fieldnames=["file", "status", "error"])
    writer.writeheader()
    for path in files:
        row = {"file": path.name, "status": "ok", "error": ""}
        try:
            process_one(path, out_dir)
        except Exception as exc:
            row = {"file": path.name, "status": "failed", "error": f"{type(exc).__name__}: {exc}"}
        writer.writerow(row)
        fh.flush()          # survive a hard kill

Report once, at the end

A batch that "keeps going" without a summary just moves the problem: now the failures are buried in 400 lines of output.

failed = [r for r in results if r["status"] == "failed"]

print("\n─── batch summary ───")
print(f"total   : {len(results)}")
print(f"ok      : {len(results) - len(failed)}")
print(f"failed  : {len(failed)}")
for r in failed:
    print(f"  ! {r['file']}: {r['error']}")

if len(failed) > len(results) * 0.1:      # your threshold, chosen deliberately
    sys.exit(1)

The exit code matters more than it looks: a scheduler, a CI job, or a Makefile decides whether the run "worked" from that number alone.

Code examples

Example 1: a complete resilient batch converter 

from pathlib import Path
import sys
import traceback
import geopandas as gpd

SRC = Path("data/raw")
OUT = Path("data/out")

def process_one(path: Path) -> Path:
    gdf = gpd.read_file(path)
    if gdf.empty:
        raise ValueError("layer has no features")
    out_path = OUT / f"{path.stem}.gpkg"
    gdf.to_file(out_path, driver="GPKG")
    return out_path

def main() -> int:
    if not SRC.is_dir():
        print(f"input folder not found: {SRC.resolve()}", file=sys.stderr)
        return 2
    files = sorted(SRC.glob("*.shp"))
    if not files:
        print(f"nothing to do in {SRC.resolve()}", file=sys.stderr)
        return 2

    OUT.mkdir(parents=True, exist_ok=True)
    ok, failed = [], []

    for i, path in enumerate(files, start=1):
        print(f"[{i}/{len(files)}] {path.name}", flush=True)
        try:
            ok.append(process_one(path))
        except Exception as exc:
            failed.append((path.name, f"{type(exc).__name__}: {exc}"))
            print(f"    failed: {exc}", file=sys.stderr, flush=True)

    print(f"\n{len(ok)} ok, {len(failed)} failed")
    for name, err in failed:
        print(f"  ! {name}: {err}")
    return 1 if failed else 0

if __name__ == "__main__":
    raise SystemExit(main())

Example 2: stop the run only after repeated failures

If every file is failing, continuing is a waste of an hour. A consecutive-failure circuit breaker gets both behaviours.

consecutive = 0
LIMIT = 5

for path in files:
    try:
        process_one(path)
        consecutive = 0
    except Exception as exc:
        consecutive += 1
        print(f"failed {path.name}: {exc}", file=sys.stderr)
        if consecutive >= LIMIT:
            print(f"aborting: {LIMIT} failures in a row", file=sys.stderr)
            break

A shared cause — an unmounted drive, an expired database password — shows up as a run of failures, not as scattered ones.

Example 3: separate expected failures from bugs

Not every exception deserves the same reaction. A missing sidecar file is data; a TypeError in your own code is a bug you want to see.

from fiona.errors import DriverError

for path in files:
    try:
        process_one(path)
    except (DriverError, ValueError) as exc:
        # data problems: record and move on
        failed.append((path.name, str(exc)))
    except Exception:
        # programming errors: stop, because every file will hit them
        print(f"unexpected error on {path.name}", file=sys.stderr)
        raise

Example 4: a failures manifest you can re-run 

import json
from pathlib import Path

Path("failures.json").write_text(
    json.dumps([{"file": n, "error": e} for n, e in failed], indent=2),
    encoding="utf-8",
)

# later, after fixing the inputs:
retry = [SRC / item["file"] for item in json.loads(Path("failures.json").read_text())]
for path in retry:
    process_one(path)

Re-running only the failures turns a two-hour batch into a two-minute one.

Explanation

An unhandled exception in Python unwinds the call stack until something catches it. A for loop is not a barrier — it is part of that stack. So an exception raised on iteration seven propagates straight out of the loop, past any code that came after it, and terminates the program. Nothing about the loop "remembers" that there were 393 iterations left.

Triage table mapping batch failure causes to their fixes.
Most batch failures are one of five things, and each has a different correct response.

Wrapping the loop in try does not help, because the exception still escapes the loop before the handler runs — you catch it, but the remaining iterations are gone. The boundary has to be inside the loop body, around one unit of work. That is the whole idea: define the unit of work, give it its own error boundary, and let the driver decide what a failure means.

Once each item is isolated, the interesting question changes from "did it crash?" to "what happened to each item?" That is why the result list matters as much as the try. A batch run has three possible outcomes — everything worked, some things worked, nothing worked — and only a per-item record can distinguish the middle case, which is by far the most common one in real GIS data.

The final piece is the exit code. Inside an automated pipeline, nobody reads your summary; a scheduler reads $?. Choosing "non-zero if any file failed" or "non-zero if more than 10% failed" is a policy decision, and making it explicitly is what separates a script that runs unattended from one that merely runs.

Edge cases or notes

  • except Exception will not catch everything: KeyboardInterrupt and SystemExit derive from BaseException, so Ctrl-C still stops the run — which is what you want. Never use bare except: to "fix" that.
  • Segfaults skip Python entirely: A crash inside GDAL or a killed process leaves no traceback. Flush a progress log after each file so a hard kill still tells you where it stopped.
  • Partial output files are still files: If to_file() fails halfway, a truncated output may exist. Write to a temporary name and rename on success so downstream steps never see a half-written file.
  • Logging beats printing for long runs: logging gives you timestamps, levels, and a file handler for free, and it is thread-safe under parallel execution.
  • Do not catch inside process_one: If the worker function swallows its own errors and returns None, the driver cannot tell success from failure. Let it raise.
  • Order affects debuggability: sorted() on the file list makes a run reproducible, so "it failed on the seventh file" means the same thing tomorrow.

FAQ

Is a broad except Exception not bad practice?

It is, everywhere except at a boundary where you have deliberately decided that one item's failure must not affect the others. The rule that makes it safe is that you record the exception rather than discard it — a broad catch with a full traceback in the log is very different from except: pass.

Should the script exit 0 or 1 when some files failed?

Exit non-zero if a human needs to look at the run. For most GIS batches that means non-zero when any file failed, because a silently skipped file becomes missing data downstream. Loosen it to a percentage threshold only when partial failure is genuinely acceptable.

How do I re-run only the files that failed?

Write the failures to a JSON or CSV manifest during the run, then feed that manifest back in as the input list. This is also the foundation of a resumable job, where the manifest of completed work lives on disk between runs.

Why does my try/except still stop the loop?

Check whether the handler calls raise, break, or sys.exit(), and whether the try is actually inside the loop body. A try wrapped around the entire for statement catches the exception but the loop has already ended by then.

Can I keep the traceback without printing it for every file?

Yes — store traceback.format_exc() in the result dictionary and print only the one-line message during the run. Dump the full tracebacks to a log file at the end, or on demand with a --verbose flag.

Does this work the same with a ThreadPoolExecutor or ProcessPoolExecutor?

Yes, and it is cleaner there: an exception raised in a worker is stored in its Future and re-raised when you call future.result(). Wrap that call in try/except and you get the same per-item boundary without changing process_one.

What if the failure is in writing the output, not reading the input?

Treat it the same way, but make sure the output directory exists before the loop starts and write via a temporary file. A permissions or disk-full error will affect every file, so a consecutive-failure circuit breaker stops the run quickly instead of grinding through 400 identical errors.