Failure Policy in Batch Processing: Fail Fast, Skip or Quarantine

Problem statement

File 47 of 900 has a corrupt geometry. What should happen?

There are only three real answers, and every batch job picks one — usually by accident:

# Policy A, chosen by writing no error handling at all
done = 0
try:
    for f in files:
        process(f)
        done += 1
except Exception as exc:
    print(f"A: died at file {done + 1} of {len(files)} ({f.name}); {len(files) - done - 1} never ran")

# Policy B, chosen by wrapping the loop in try/except and moving on
written = 0
for f in files:
    try:
        process(f)
        written += 1
    except Exception:
        pass
print(f"B: {written} processed, {len(files) - written} silently missing")

# Policy C, chosen deliberately
quarantined = []
for f in files:
    try:
        process(f)
    except Exception as exc:
        quarantined.append((f.name, f"{type(exc).__name__}: {exc}"))    # set aside with its reason
print(f"C: {len(files) - len(quarantined)} processed, {len(quarantined)} set aside:")
for name, why in quarantined:
    print(f"   {name:10} {why[:70]}")

A: died at file 1 of 18 (coast.shp); 17 never ran
B: 14 processed, 4 silently missing
C: 14 processed, 4 set aside:
   coast.shp  DataSourceError: Unable to open data/raw/coast.shx or data/raw/coast.S
   lakes.shp  ValueError: no CRS
   land.shp   DataSourceError: 'data/raw/land.shp' not recognized as being in a supp
   rivers.shp ValueError: 6 of 13 geometries missing

Eighteen real shapefiles, five of them damaged the ways deliveries arrive (see The data used on this page). Policy A never gets past the first file, because sorting puts the one without a .shx first. Policy B reports nothing and loses four. Policy C loses the same four and says which and why — and even C lets one damaged file through: the ocean without its .dbf reads as geometry with no attributes, and nothing in process asks for them. A policy decides what happens to the failures you detect; it cannot detect them for you.

Policy B is the most common and the most dangerous. The job exits zero, the output folder looks full, and the one missing file surfaces three weeks later when a total does not reconcile.

The decision is not "should I handle errors" — it is which failures should stop the run, and what should happen to the items that fail. Those are two different questions, and conflating them is why most batch error handling is wrong.

Quick answer

Classify the failure first, then apply a policy:

class Fatal(Exception):
    """Nothing will succeed after this. Stop the run."""

def guarded(fn, item):
    try:
        return Result(item, "ok", value=fn(item))
    except Fatal:
        raise                                    # environmental — let it stop the run
    except (OSError, MemoryError) as exc:
        raise Fatal(f"{item}: {exc}") from exc   # promote: these do not get better
    except Exception as exc:
        return Result(item, "failed", f"{type(exc).__name__}: {exc}")

def run(items, fn, *, abort_after=None):
    results, failures = [], 0
    for item in items:
        r = guarded(fn, item)
        results.append(r)
        if r.status == "failed":
            failures += 1
            if abort_after and failures >= abort_after:
                raise Fatal(f"{failures} failures — stopping, this looks systematic")
    return results

Three rules that make the difference:

1. A bare except Exception: pass is never the answer. Record it or raise it.
2. The exit code must reflect reality. Non-zero if anything failed, or nothing downstream can tell.
3. A rising failure count is itself a signal. Twenty failures in a row is not twenty per-item problems.

The three policies

This item or this run: quarantine, abort on a pattern

1. Distinguish "this item" from "this run"

The question that classifies a failure is: does it predict anything about the other items?

FATAL_TYPES = (
    MemoryError,           # the next file is probably bigger
    KeyboardInterrupt,     # you asked it to stop
    SystemExit,
)

FATAL_MESSAGES = (
    "no space left on device",
    "could not connect to server",
    "permission denied",
    "too many open files",
)

A disk that is full will be full for item 48. A database that is down will be down. A corrupt shapefile says nothing whatsoever about the next one. That is the whole distinction, and it is worth encoding rather than remembering.

def is_fatal(exc: Exception) -> bool:
    if isinstance(exc, FATAL_TYPES):
        return True
    msg = str(exc).lower()
    return any(m in msg for m in FATAL_MESSAGES)

Message matching is imperfect. Prefer catching the specific exception type where the library offers one, and treat the message list as a safety net rather than the mechanism.

2. Quarantine, do not discard

A quarantined item is set aside with enough context to diagnose it later, without blocking the run.

def quarantine(item: Path, exc: Exception, quarantine_dir: Path) -> None:
    quarantine_dir.mkdir(parents=True, exist_ok=True)
    record = {
        "item": str(item),
        "error": f"{type(exc).__name__}: {exc}",
        "traceback": traceback.format_exc(),
        "when": run_started_at.isoformat(),
    }
    (quarantine_dir / f"{item.stem}.json").write_text(json.dumps(record, indent=2))

What makes quarantine different from skipping is that the item ends up somewhere a human will look. Three properties are worth having:

• The reason is recorded, not just the fact of failure.
• The item is re-runnable once fixed, without re-running the whole batch.
• The count is visible in the summary, so nobody has to go looking.

Copying the offending file into the quarantine directory is often worth the disk — a supplier who sends a corrupt file will replace it, and having the original makes the conversation short.

3. Abort on a pattern, not on a single failure

One failure in 900 is a bad file. Two hundred failures in 900 means the assumption is wrong — a schema change, a supplier switching CRS, an upstream export that half-completed.

def run(items, fn, *, abort_after=20, abort_ratio=0.25):
    results, failures = [], 0
    for i, item in enumerate(items, start=1):
        r = guarded(fn, item)
        results.append(r)
        if r.status != "failed":
            continue
        failures += 1
        if failures >= abort_after:
            raise Fatal(f"{failures} failures — aborting, this is not per-item")
        if i >= 50 and failures / i > abort_ratio:
            raise Fatal(f"{failures}/{i} failing ({failures/i:.0%}) — aborting")
    return results

Both thresholds earn their place. The absolute count catches a systematic problem in a small run; the ratio catches one in a large run without waiting for twenty failures out of ten thousand.

4. Make the exit code tell the truth

def main() -> int:
    results = run(discover(SRC, DST), apply_one)
    summary = report(results)
    print(f"{summary['ok']} ok · {summary['failed']} failed")

    if summary["failed"]:
        return 1        # a scheduler, CI job or wrapper can now see it
    return 0

if __name__ == "__main__":
    sys.exit(main())

A job that catches every exception and exits zero is invisible to everything monitoring it. Cron sends no mail, CI goes green, the alerting rule never fires. See my GIS pipeline fails silently.

There is a legitimate middle setting — exit zero when failures are within an expected tolerance:

tolerated = summary["items"] * 0.01           # 1% of items may fail
return 0 if summary["failed"] <= tolerated else 1

Explicit, documented, and reported either way. That is different from swallowing.

5. Choose per-step, not per-job

One job can want different policies at different points:

def apply_one(item: Path) -> Result:
    gdf = read_or_fatal(item)                  # unreadable → quarantine this item
    gdf = validate_or_fatal(gdf)               # wrong schema → FATAL, all files share it
    gdf = clean(gdf)                           # bad geometry → repair, log, continue
    write_or_fatal(gdf, output_for(item))      # disk full → FATAL
    return Result(item, "ok", rows=len(gdf))

The read failure is per-item; the schema failure is systematic; the write failure is environmental. Encoding that inside apply_one keeps the classification next to the operation that knows what it means.

A policy-driven runner with quarantine re-runs

The data used on this page

A real folder of shapefiles: thirteen layers of Natural Earth (public domain) at 1:110 million and 1:50 million, flat in data/raw/. Five more are copies damaged on purpose, each in a way deliveries really arrive: lakes.shp without its .prj; rivers.shp cut off halfway, as an interrupted copy leaves it; coast.shp without its .shx; ocean.shp without its .dbf; and land.shp, which is a web server's 404 page saved under a shapefile's name.

import io
import shutil
import zipfile
from pathlib import Path

import requests

NE = {"cultural": ["ne_110m_admin_0_countries", "ne_110m_admin_1_states_provinces", "ne_110m_populated_places",
                   "ne_110m_admin_0_boundary_lines_land", "ne_50m_admin_0_countries", "ne_50m_populated_places"],
      "physical": ["ne_110m_coastline", "ne_110m_land", "ne_110m_ocean", "ne_110m_rivers_lake_centerlines",
                   "ne_110m_lakes", "ne_110m_glaciated_areas", "ne_110m_geography_regions_polys"]}
SRC = Path("data/ne")
for d in ("data/raw", "data/out"):
    shutil.rmtree(d, ignore_errors=True)               # every run of the page starts from nothing
for theme, names in NE.items():
    for name in names:
        if not (SRC / theme / f"{name}.shp").exists():
            scale = name.split("_")[1]
            reply = requests.get(f"https://naciscdn.org/naturalearth/{scale}/{theme}/{name}.zip", timeout=120)
            reply.raise_for_status()
            zipfile.ZipFile(io.BytesIO(reply.content)).extractall(SRC / theme)

raw = Path("data/raw")
raw.mkdir(parents=True)
for f in SRC.rglob("*"):
    if f.suffix in (".shp", ".shx", ".dbf", ".prj", ".cpg"):
        shutil.copy(f, raw / f.name)

# five deliveries damaged on purpose, each the way files really arrive damaged
def deliver(layer, name, parts):
    for ext in parts:
        shutil.copy(SRC / "physical" / f"ne_110m_{layer}.{ext}", raw / f"{name}.{ext}")

deliver("lakes", "lakes", ["shp", "shx", "dbf"])                      # no .prj
deliver("rivers_lake_centerlines", "rivers", ["shp", "shx", "dbf", "prj"])
data = (raw / "rivers.shp").read_bytes()
(raw / "rivers.shp").write_bytes(data[: len(data) // 2])              # cut off mid-copy
deliver("coastline", "coast", ["shp", "dbf", "prj"])                  # no .shx
deliver("ocean", "ocean", ["shp", "shx", "prj"])                      # no .dbf
deliver("land", "land", ["shx", "dbf", "prj"])
(raw / "land.shp").write_text("<html><body><h1>404 Not Found</h1></body></html>")

# the job every example on this page runs: reproject a layer to Equal Earth and write it
import json
import traceback
from dataclasses import dataclass
from datetime import datetime

import geopandas as gpd

@dataclass
class Result:
    item: str
    status: str
    detail: str = ""
    value: object = None

run_started_at = datetime.now()
files = sorted(raw.glob("*.shp"))
out = Path("data/out")
out.mkdir(parents=True, exist_ok=True)

def process(path):
    gdf = gpd.read_file(path)
    if gdf.crs is None:
        raise ValueError("no CRS")
    if gdf.geometry.isna().any():
        raise ValueError(f"{gdf.geometry.isna().sum()} of {len(gdf)} geometries missing")
    gdf.to_crs(8857).to_file(out / f"{Path(path).stem}.gpkg", driver="GPKG")
    return len(gdf)

print(len(files), "shapefiles in", raw)

18 shapefiles in data/raw

Example 1: a policy-driven runner

from dataclasses import dataclass
from enum import Enum

class OnError(str, Enum):
    FAIL_FAST = "fail-fast"     # stop at the first failure
    SKIP = "skip"               # record and continue
    QUARANTINE = "quarantine"   # record, copy aside, continue

def run(items, fn, *, on_error=OnError.QUARANTINE, quarantine_dir=None,
        abort_after=20):
    results, failures = [], 0
    for item in items:
        try:
            results.append(fn(item))
            continue
        except Exception as exc:
            if is_fatal(exc) or on_error is OnError.FAIL_FAST:
                raise
            failures += 1
            if on_error is OnError.QUARANTINE and quarantine_dir:
                quarantine(item, exc, quarantine_dir)
            results.append(Result(str(item), "failed", f"{type(exc).__name__}: {exc}"))
            if failures >= abort_after:
                raise Fatal(f"{failures} failures — aborting")
    return results

def apply_one(item):
    return Result(str(item), "ok", value=process(item))

shutil.rmtree("quarantine", ignore_errors=True)
results = run(files, apply_one, on_error=OnError.QUARANTINE, quarantine_dir=Path("quarantine"))
print(sum(r.status == "ok" for r in results), "ok;", sorted(p.name for p in Path("quarantine").glob("*.json")))

14 ok; ['coast.json', 'lakes.json', 'land.json', 'rivers.json']

fail-fast is the right default while developing, quarantine in production. Making it a flag means you do not have to edit code to switch:

python -m pipeline run --on-error fail-fast     # while debugging
python -m pipeline run --on-error quarantine    # overnight

Example 2: re-running only the quarantined items

def rerun_quarantined(quarantine_dir: Path, fn) -> list[Result]:
    records = [json.loads(p.read_text()) for p in quarantine_dir.glob("*.json")]
    items = [Path(r["item"]) for r in records]
    print(f"retrying {len(items)} quarantined items")

    results = run(items, fn, on_error=OnError.QUARANTINE, quarantine_dir=quarantine_dir)
    for r in results:
        if r.status == "ok":
            (quarantine_dir / f"{Path(r.item).stem}.json").unlink(missing_ok=True)
    return results

# fix what can be fixed: rebuild coast's .shx, fetch land again, give the lakes their .prj
from pyogrio import set_gdal_config_options
set_gdal_config_options({"SHAPE_RESTORE_SHX": "YES"})
gpd.read_file(raw / "coast.shp")
set_gdal_config_options({"SHAPE_RESTORE_SHX": None})
shutil.copy(SRC / "physical" / "ne_110m_land.shp", raw / "land.shp")
shutil.copy(SRC / "physical" / "ne_110m_lakes.prj", raw / "lakes.prj")

again = rerun_quarantined(Path("quarantine"), apply_one)
print("still quarantined:", sorted(p.stem for p in Path("quarantine").glob("*.json")))

retrying 4 quarantined items
still quarantined: ['rivers']

Three of the four came back once their causes were fixed: the index rebuilt, the 404 page fetched again, the .prj supplied. The rivers stay quarantined because the file itself is cut short and no retry will lengthen it — which is what quarantine is for: the record is still there, with its reason, for someone to request a new copy.

The quarantine directory doubles as the retry queue. Clearing an entry on success means the directory is always "what is still broken" rather than a growing history — which is what makes anyone actually look at it.

Example 3: failure classes in the summary

from collections import Counter

def failure_breakdown(results) -> dict[str, int]:
    """Twenty failures of one kind is a different problem from twenty of twenty."""
    return Counter(
        r.detail.split(":")[0] for r in results if r.status == "failed"
    )

print(failure_breakdown(results))

Counter({'DataSourceError': 2, 'ValueError': 2})

Counting by exception type is a start, and here it is a coarse one: "no CRS" and "6 of 13 geometries missing" are both ValueError. Give your own checks their own exception classes, or count by the message, so the breakdown separates problems that need different fixes.

Eighteen identical failures is a systematic problem wearing a per-item costume. Grouping by exception type in the summary is the cheapest way to see it — and a good trigger for tightening abort_after.

Why except-pass defers the cost

The reason except Exception: pass is so common is that it makes the immediate problem go away: the job finishes. The cost is deferred and invisible, which is the worst combination — the run reports success, the output directory has files in it, and the only evidence of the missing 47th file is a number somewhere downstream that is slightly too small.

What quarantine adds is an audit trail with a shape. After the run there are three sets: succeeded, skipped-because-already-done, and failed-with-a-reason. Those three sum to the discovered item count, and that arithmetic is checkable:

counts = Counter(r.status for r in results)
summary = {"items": len(files), "ok": counts["ok"], "skipped": counts["skipped"], "failed": counts["failed"]}
assert summary["ok"] + summary["skipped"] + summary["failed"] == summary["items"]
print(summary)

{'items': 18, 'ok': 14, 'skipped': 0, 'failed': 4}

A batch job where that assertion holds can answer "what happened to file X" for every X. One with a bare pass cannot answer it for any X.

The abort-on-pattern rule deserves its own justification. A single per-item failure is genuinely per-item and should not stop 899 other files from processing. But failures are rarely independent — a supplier changes their export settings and every file that day is affected. Without a threshold, the job dutifully quarantines 900 files, reports 900 failures, and burns the whole overnight window producing nothing. The threshold converts "the job worked and everything failed" into "the job stopped and told you why", which is the outcome you want at 3am.

Finally, the exit code is not a detail. Cron mails on non-zero output, CI fails on non-zero exit, an orchestrator retries or alerts on non-zero. A job that always exits zero has opted out of every one of those mechanisms, no matter how good its logging is — because nothing is reading the log until something tells it to. See how to get alerted when an automated GIS job fails.

BaseException, worker failures and retries

• except Exception does not catch KeyboardInterrupt or SystemExit — they inherit from BaseException. That is correct behaviour: Ctrl-C should stop the run.
• Catching BaseException makes a job unkillable. Never do it in a loop.
• A failure inside a parallel worker must be returned, not raised, or the pool's behaviour depends on the executor. Return a Result with status="failed".
• Retries and failure policy are different things. Retry a transient failure (network, lock) two or three times; quarantine a deterministic one immediately — retrying a corrupt file 3× just wastes time. See how to add retries and timeouts.
• OSError covers both "disk full" and "file not found" — the first is fatal, the second per-item. Check errno rather than the type alone.
• Quarantine directories grow. Clear on success, and age out entries, or nobody will look at it after a month.
• A partially written output file is worse than none. Write to a temp path and rename on success, so a failure never leaves a plausible-looking truncated file.
• In tests, use fail-fast. A test suite that quarantines failures is a test suite that passes.

Internal links

• The anatomy of a batch job: discover, apply, report — where the policy sits in the structure
• Batch script stops at the first bad file — the practical fix for policy A
• How to log and summarise errors in a batch GIS job — recording what quarantine captures
• My GIS pipeline fails silently — what a wrong exit code costs
• How to add retries and timeouts to an automated GIS job — the policy for transient failures
• How to get alerted when an automated GIS job fails — who finds out, and how
• How to build a resumable batch GIS job in Python — re-running what failed
• Logs, metrics and alerts: observability for GIS pipelines — the wider picture

FAQ

What is wrong with except Exception: pass?

It converts a failure into a silence. The run reports success, the output looks complete, and there is no record of which item is missing or why. Record it or raise it — never neither.

Should a batch job ever stop on the first error?

While developing, yes — fail-fast gives you a traceback at the point of damage. In production it usually should not, unless the failure is environmental or systematic.

How many failures should trigger an abort?

Twenty, or 25% of items processed so far, whichever comes first. Both matter: the count catches systematic problems in small runs, the ratio catches them in large ones.

What is the difference between skipping and quarantining?

Skipping records that an item was not processed. Quarantining also records why, keeps the item retryable, and puts it somewhere a person will look.

Should the job exit non-zero if one file failed?

Yes by default. If a small failure rate is genuinely acceptable, encode that as an explicit tolerance and still report the count — do not achieve it by exiting zero unconditionally.

How do I handle failures in a parallel run?

Catch inside the worker and return a Result object. Raising inside a pool worker makes the outcome depend on the executor, and can lose the traceback.

Where should the failure classification live?

Next to the operation that knows what the failure means. A read failure is per-item; a schema validation failure is systematic. apply_one is the right place to make that call.