🧭 Concept Batch processing Pipelines & automation Updated

Failure Policy in Batch Processing: Fail Fast, Skip or Quarantine

Problem statement

File 47 of 900 has a corrupt geometry. What should happen?

There are only three real answers, and every batch job picks one — usually by accident:

# Policy A, chosen by writing no error handling at all
for f in files:
    process(f)                      # dies at 47, files 48-900 never run

# Policy B, chosen by wrapping the loop in try/except and moving on
for f in files:
    try:
        process(f)
    except Exception:
        pass                        # 899 files processed, one silently missing

# Policy C, chosen deliberately
for f in files:
    try:
        process(f)
    except Exception as exc:
        quarantine(f, exc)          # 899 processed, one set aside with its reason

Policy B is the most common and the most dangerous. The job exits zero, the output folder looks full, and the one missing file surfaces three weeks later when a total does not reconcile.

The decision is not "should I handle errors" — it is which failures should stop the run, and what should happen to the items that fail. Those are two different questions, and conflating them is why most batch error handling is wrong.

Quick answer

Classify the failure first, then apply a policy:

Failure kind Example Policy
Environmental disk full, database down, no permissions fail fast — nothing will succeed
Systematic every file missing a CRS, wrong schema fail fast — the assumption is wrong
Per-item one corrupt file, one bad geometry quarantine — set aside, keep going
Expected file already processed, empty input skip — not a failure at all 
class Fatal(Exception):
    """Nothing will succeed after this. Stop the run."""

def guarded(fn, item):
    try:
        return Result(item, "ok", value=fn(item))
    except Fatal:
        raise                                    # environmental — let it stop the run
    except (OSError, MemoryError) as exc:
        raise Fatal(f"{item}: {exc}") from exc   # promote: these do not get better
    except Exception as exc:
        return Result(item, "failed", f"{type(exc).__name__}: {exc}")

def run(items, fn, *, abort_after=None):
    results, failures = [], 0
    for item in items:
        r = guarded(fn, item)
        results.append(r)
        if r.status == "failed":
            failures += 1
            if abort_after and failures >= abort_after:
                raise Fatal(f"{failures} failures — stopping, this looks systematic")
    return results

Three rules that make the difference:

  1. A bare except Exception: pass is never the answer. Record it or raise it.
  2. The exit code must reflect reality. Non-zero if anything failed, or nothing downstream can tell.
  3. A rising failure count is itself a signal. Twenty failures in a row is not twenty per-item problems.

The three policies

Three panels comparing fail fast, silent skip and quarantine outcomes for a 900-file run.
Same corrupt file, three outcomes. Only one leaves you able to answer what happened.

Step-by-step solution

Decision diagram routing a failure to fail fast, quarantine or skip based on its kind.
Ask what else this failure predicts. That answer is the policy.

1. Distinguish "this item" from "this run"

The question that classifies a failure is: does it predict anything about the other items?

FATAL_TYPES = (
    MemoryError,           # the next file is probably bigger
    KeyboardInterrupt,     # you asked it to stop
    SystemExit,
)

FATAL_MESSAGES = (
    "no space left on device",
    "could not connect to server",
    "permission denied",
    "too many open files",
)

A disk that is full will be full for item 48. A database that is down will be down. A corrupt shapefile says nothing whatsoever about the next one. That is the whole distinction, and it is worth encoding rather than remembering.

def is_fatal(exc: Exception) -> bool:
    if isinstance(exc, FATAL_TYPES):
        return True
    msg = str(exc).lower()
    return any(m in msg for m in FATAL_MESSAGES)

Message matching is imperfect. Prefer catching the specific exception type where the library offers one, and treat the message list as a safety net rather than the mechanism.

2. Quarantine, do not discard

A quarantined item is set aside with enough context to diagnose it later, without blocking the run.

def quarantine(item: Path, exc: Exception, quarantine_dir: Path) -> None:
    quarantine_dir.mkdir(parents=True, exist_ok=True)
    record = {
        "item": str(item),
        "error": f"{type(exc).__name__}: {exc}",
        "traceback": traceback.format_exc(),
        "when": run_started_at.isoformat(),
    }
    (quarantine_dir / f"{item.stem}.json").write_text(json.dumps(record, indent=2))

What makes quarantine different from skipping is that the item ends up somewhere a human will look. Three properties are worth having:

  • The reason is recorded, not just the fact of failure.
  • The item is re-runnable once fixed, without re-running the whole batch.
  • The count is visible in the summary, so nobody has to go looking.

Copying the offending file into the quarantine directory is often worth the disk — a supplier who sends a corrupt file will replace it, and having the original makes the conversation short.

3. Abort on a pattern, not on a single failure

One failure in 900 is a bad file. Two hundred failures in 900 means the assumption is wrong — a schema change, a supplier switching CRS, an upstream export that half-completed.

def run(items, fn, *, abort_after=20, abort_ratio=0.25):
    results, failures = [], 0
    for i, item in enumerate(items, start=1):
        r = guarded(fn, item)
        results.append(r)
        if r.status != "failed":
            continue
        failures += 1
        if failures >= abort_after:
            raise Fatal(f"{failures} failures — aborting, this is not per-item")
        if i >= 50 and failures / i > abort_ratio:
            raise Fatal(f"{failures}/{i} failing ({failures/i:.0%}) — aborting")
    return results

Both thresholds earn their place. The absolute count catches a systematic problem in a small run; the ratio catches one in a large run without waiting for twenty failures out of ten thousand.

4. Make the exit code tell the truth 

def main() -> int:
    results = run(discover(SRC, DST), apply_one)
    summary = report(results)
    print(f"{summary['ok']} ok · {summary['failed']} failed")

    if summary["failed"]:
        return 1        # a scheduler, CI job or wrapper can now see it
    return 0

if __name__ == "__main__":
    sys.exit(main())

A job that catches every exception and exits zero is invisible to everything monitoring it. Cron sends no mail, CI goes green, the alerting rule never fires. See my GIS pipeline fails silently.

There is a legitimate middle setting — exit zero when failures are within an expected tolerance:

tolerated = summary["items"] * 0.01           # 1% of items may fail
return 0 if summary["failed"] <= tolerated else 1

Explicit, documented, and reported either way. That is different from swallowing.

5. Choose per-step, not per-job

One job can want different policies at different points:

def apply_one(item: Path) -> Result:
    gdf = read_or_fatal(item)                  # unreadable → quarantine this item
    gdf = validate_or_fatal(gdf)               # wrong schema → FATAL, all files share it
    gdf = clean(gdf)                           # bad geometry → repair, log, continue
    write_or_fatal(gdf, output_for(item))      # disk full → FATAL
    return Result(item, "ok", rows=len(gdf))

The read failure is per-item; the schema failure is systematic; the write failure is environmental. Encoding that inside apply_one keeps the classification next to the operation that knows what it means.

Code examples

Example 1: a policy-driven runner 

from dataclasses import dataclass
from enum import Enum

class OnError(str, Enum):
    FAIL_FAST = "fail-fast"     # stop at the first failure
    SKIP = "skip"               # record and continue
    QUARANTINE = "quarantine"   # record, copy aside, continue

def run(items, fn, *, on_error=OnError.QUARANTINE, quarantine_dir=None,
        abort_after=20):
    results, failures = [], 0
    for item in items:
        try:
            results.append(fn(item))
            continue
        except Exception as exc:
            if is_fatal(exc) or on_error is OnError.FAIL_FAST:
                raise
            failures += 1
            if on_error is OnError.QUARANTINE and quarantine_dir:
                quarantine(item, exc, quarantine_dir)
            results.append(Result(str(item), "failed", f"{type(exc).__name__}: {exc}"))
            if failures >= abort_after:
                raise Fatal(f"{failures} failures — aborting")
    return results

fail-fast is the right default while developing, quarantine in production. Making it a flag means you do not have to edit code to switch:

python -m pipeline run --on-error fail-fast     # while debugging
python -m pipeline run --on-error quarantine    # overnight

Example 2: re-running only the quarantined items 

def rerun_quarantined(quarantine_dir: Path, fn) -> list[Result]:
    records = [json.loads(p.read_text()) for p in quarantine_dir.glob("*.json")]
    items = [Path(r["item"]) for r in records]
    print(f"retrying {len(items)} quarantined items")

    results = run(items, fn, on_error=OnError.QUARANTINE, quarantine_dir=quarantine_dir)
    for r in results:
        if r.status == "ok":
            (quarantine_dir / f"{Path(r.item).stem}.json").unlink(missing_ok=True)
    return results

The quarantine directory doubles as the retry queue. Clearing an entry on success means the directory is always "what is still broken" rather than a growing history — which is what makes anyone actually look at it.

Example 3: failure classes in the summary 

def failure_breakdown(results) -> dict[str, int]:
    """Twenty failures of one kind is a different problem from twenty of twenty."""
    return Counter(
        r.detail.split(":")[0] for r in results if r.status == "failed"
    )

print(failure_breakdown(results))
# {'CRSError': 18, 'DriverError': 1, 'TopologyException': 1}

Eighteen identical failures is a systematic problem wearing a per-item costume. Grouping by exception type in the summary is the cheapest way to see it — and a good trigger for tightening abort_after.

Explanation

Checklist showing what a bare except-pass costs compared with a recorded failure.
Every line here is something the run can no longer tell you.

The reason except Exception: pass is so common is that it makes the immediate problem go away: the job finishes. The cost is deferred and invisible, which is the worst combination — the run reports success, the output directory has files in it, and the only evidence of the missing 47th file is a number somewhere downstream that is slightly too small.

What quarantine adds is an audit trail with a shape. After the run there are three sets: succeeded, skipped-because-already-done, and failed-with-a-reason. Those three sum to the discovered item count, and that arithmetic is checkable:

assert summary["ok"] + summary["skipped"] + summary["failed"] == summary["items"]

A batch job where that assertion holds can answer "what happened to file X" for every X. One with a bare pass cannot answer it for any X.

The abort-on-pattern rule deserves its own justification. A single per-item failure is genuinely per-item and should not stop 899 other files from processing. But failures are rarely independent — a supplier changes their export settings and every file that day is affected. Without a threshold, the job dutifully quarantines 900 files, reports 900 failures, and burns the whole overnight window producing nothing. The threshold converts "the job worked and everything failed" into "the job stopped and told you why", which is the outcome you want at 3am.

Finally, the exit code is not a detail. Cron mails on non-zero output, CI fails on non-zero exit, an orchestrator retries or alerts on non-zero. A job that always exits zero has opted out of every one of those mechanisms, no matter how good its logging is — because nothing is reading the log until something tells it to. See how to get alerted when an automated GIS job fails.

Edge cases or notes

  • except Exception does not catch KeyboardInterrupt or SystemExit — they inherit from BaseException. That is correct behaviour: Ctrl-C should stop the run.
  • Catching BaseException makes a job unkillable. Never do it in a loop.
  • A failure inside a parallel worker must be returned, not raised, or the pool's behaviour depends on the executor. Return a Result with status="failed".
  • Retries and failure policy are different things. Retry a transient failure (network, lock) two or three times; quarantine a deterministic one immediately — retrying a corrupt file 3× just wastes time. See how to add retries and timeouts.
  • OSError covers both "disk full" and "file not found" — the first is fatal, the second per-item. Check errno rather than the type alone.
  • Quarantine directories grow. Clear on success, and age out entries, or nobody will look at it after a month.
  • A partially written output file is worse than none. Write to a temp path and rename on success, so a failure never leaves a plausible-looking truncated file.
  • In tests, use fail-fast. A test suite that quarantines failures is a test suite that passes.

FAQ

What is wrong with except Exception: pass?

It converts a failure into a silence. The run reports success, the output looks complete, and there is no record of which item is missing or why. Record it or raise it — never neither.

Should a batch job ever stop on the first error?

While developing, yes — fail-fast gives you a traceback at the point of damage. In production it usually should not, unless the failure is environmental or systematic.

How many failures should trigger an abort?

Twenty, or 25% of items processed so far, whichever comes first. Both matter: the count catches systematic problems in small runs, the ratio catches them in large ones.

What is the difference between skipping and quarantining?

Skipping records that an item was not processed. Quarantining also records why, keeps the item retryable, and puts it somewhere a person will look.

Should the job exit non-zero if one file failed?

Yes by default. If a small failure rate is genuinely acceptable, encode that as an explicit tolerance and still report the count — do not achieve it by exiting zero unconditionally.

How do I handle failures in a parallel run?

Catch inside the worker and return a Result object. Raising inside a pool worker makes the outcome depend on the executor, and can lose the traceback.

Where should the failure classification live?

Next to the operation that knows what the failure means. A read failure is per-item; a schema validation failure is systematic. apply_one is the right place to make that call.