🔧 Fix Pipelines & automation Automation & Pipelines · step 26 of 26 Updated

My GIS Pipeline Fails Silently: Fixing Swallowed Errors and Wrong Exit Codes

Problem statement

The scheduler reports a green run every night. Three weeks later someone notices the output layer has been empty since the 14th.

$ tail -3 logs/nightly.log
[2026-08-11 02:31:07] starting nightly parcels build
[2026-08-11 02:31:44] done
$ echo $?
0

Nothing crashed, so nothing alerted. But "done" was printed by a script that caught an exception, wrote a zero-row file, and exited 0 anyway. The pipeline did not fail — it succeeded at producing nothing, which is much harder to notice.

Common causes:

  • except Exception: pass, or an except block that logs at DEBUG level
  • an exit code of 0 returned regardless of what happened
  • a shell wrapper where only the last command's status is checked (cmd1; cmd2)
  • a pipeline a | b whose exit status reflects b alone
  • output written before validation, so a truncated result looks like a result
  • an empty input folder treated as "nothing to do" rather than as a problem
  • a try around a whole stage, so a failure skips work without stopping the run

The unifying theme: at every boundary — function, process, shell, scheduler — there is a signal that says "this went wrong", and each of those signals can be dropped independently.

Quick answer

To make failures visible:

  1. never write except: pass; log with logger.exception() and re-raise or record
  2. return a real exit code — non-zero on any failure — from main()
  3. use set -euo pipefail in every shell wrapper
  4. assert on the result, not just the absence of exceptions: row counts, CRS, bbox, file size
  5. emit a machine-readable run summary and alert on it, including "did the job run at all"
import logging, sys

log = logging.getLogger("pipeline")

def main() -> int:
    try:
        result = run_pipeline()
    except Exception:
        log.exception("pipeline failed")          # full traceback at ERROR level
        return 1

    if result["features"] == 0:
        log.error("pipeline produced 0 features — treating as failure")
        return 1

    log.info("wrote %s features to %s", result["features"], result["output"])
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

raise SystemExit(main()) is the whole exit-code contract in one line: whatever main returns becomes the process status the scheduler sees.

Where the signal gets dropped

A chain from exception to alert with five places the failure signal can be lost.
Five hand-offs, five chances to lose the failure — one break makes the run look green.

Step-by-step solution

Grid of exit code conventions and what each means to a scheduler.
An exit code is an API. Decide what each value means, and be consistent.

Stop swallowing exceptions

There are three legitimate reasons to catch an exception: to add context and re-raise, to record a per-item failure in a batch, or to handle a genuinely expected condition. Everything else hides bugs.

# hides everything, including typos in your own code
try:
    gdf = gpd.read_file(path)
except Exception:
    pass

# records the failure and keeps the batch alive — the pattern for per-file loops
try:
    gdf = gpd.read_file(path)
except Exception as exc:
    failures.append((path.name, f"{type(exc).__name__}: {exc}"))
    log.warning("skipping %s: %s", path.name, exc)

# adds context and re-raises — the pattern for a single-step pipeline
try:
    gdf = gpd.read_file(path)
except Exception as exc:
    raise RuntimeError(f"could not read {path}") from exc

raise ... from exc keeps the original traceback attached, so the log shows both the context and the cause.

Log exceptions at a level someone reads

A traceback logged at DEBUG in a job configured at INFO is a traceback nobody will ever see.

import logging, sys
from pathlib import Path

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)-8s %(name)s %(message)s",
    handlers=[logging.FileHandler(Path("logs/pipeline.log"), encoding="utf-8"),
              logging.StreamHandler(sys.stdout)],
)

try:
    step()
except Exception:
    logging.getLogger("pipeline").exception("step failed")   # ERROR + traceback
    raise

logger.exception() is logger.error() plus the traceback, and it only works inside an except block — which is exactly where you want it.

Return an honest exit code

Schedulers, CI and make all decide success from the process status. A script that always exits 0 cannot be monitored.

EXIT_OK          = 0
EXIT_FAILED      = 1     # the work was attempted and did not succeed
EXIT_BAD_CONFIG  = 2     # refused to start: bad input, missing path, bad config
EXIT_NO_INPUT    = 3     # nothing to process — decide whether this is an error

def main() -> int:
    try:
        cfg = load_config(sys.argv[1])
    except (FileNotFoundError, ValueError) as exc:
        print(f"config error: {exc}", file=sys.stderr)
        return EXIT_BAD_CONFIG
    ...

Distinct codes let a wrapper react differently: retry a transient failure, page someone for a configuration error, ignore "no input" on a public holiday.

Make the shell wrapper strict

By default, a shell script continues after a failing command and reports the status of the last one.

#!/usr/bin/env bash
set -euo pipefail        # -e: exit on error, -u: undefined vars are errors,
                         # -o pipefail: a failing stage fails the whole pipe

python extract.py
python transform.py
python load.py

Without -e, a failing extract.py still runs transform.py on stale data and the wrapper exits 0. Without pipefail, python job.py | tee log.txt reports tee's status — always 0.

Where you need to continue deliberately, be explicit:

if ! python optional_step.py; then
  echo "optional step failed — continuing" >&2
fi

Validate the output, not just the absence of errors

The most dangerous failures raise nothing at all. Check the result against expectations before declaring success.

def assert_output_sane(gdf, previous_count: int | None = None) -> None:
    if len(gdf) == 0:
        raise ValueError("output has zero features")
    if gdf.crs is None:
        raise ValueError("output has no CRS")
    if gdf.geometry.isna().any():
        raise ValueError(f"{gdf.geometry.isna().sum()} null geometries in output")
    if not gdf.geometry.is_valid.all():
        raise ValueError(f"{(~gdf.geometry.is_valid).sum()} invalid geometries in output")
    if previous_count and len(gdf) < previous_count * 0.5:
        raise ValueError(f"feature count halved: {previous_count} → {len(gdf)}")

Comparing against the previous run catches the class of failure where an upstream source quietly starts returning a fraction of its data — no error, just less of it.

Write a run summary a machine can read

Human-readable logs are for debugging; a structured summary is for alerting.

import json
from datetime import datetime, timezone
from pathlib import Path

def write_summary(path: Path, **fields) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    record = {"finished_at": datetime.now(timezone.utc).isoformat(), **fields}
    path.write_text(json.dumps(record, indent=2), encoding="utf-8")

write_summary(Path("logs/last_run.json"),
              status="ok", features=len(gdf), duration_s=round(elapsed, 1),
              output=str(out_path), input_files=len(files), failures=len(failed))

A monitor that reads last_run.json can alert on status != "ok", on a feature count that dropped, and on a finished_at that is older than expected — which is the only way to detect a job that never started.

Detect the job that did not run

A silent failure includes silence itself. Dead-man's-switch monitoring inverts the logic: alert when the success signal is missing.

import requests   # ping a heartbeat endpoint on success

def heartbeat(url: str) -> None:
    try:
        requests.get(url, timeout=10)
    except Exception:
        log.warning("heartbeat failed — not fatal")

if status == "ok":
    heartbeat("https://hc-ping.com/<uuid>")

Any hosted cron-monitoring service works this way, and a five-line check against last_run.json mtime does the same job without a dependency.

Code examples

Example 1: a pipeline entry point with a real contract 

from pathlib import Path
import logging, sys, time
import geopandas as gpd

log = logging.getLogger("nightly")

def run(cfg) -> dict:
    started = time.perf_counter()
    files = sorted(Path(cfg["input_dir"]).glob("*.gpkg"))
    if not files:
        raise FileNotFoundError(f"no input files in {cfg['input_dir']}")

    frames, failures = [], []
    for path in files:
        try:
            frames.append(gpd.read_file(path))
        except Exception as exc:
            failures.append((path.name, str(exc)))
            log.warning("skipping %s: %s", path.name, exc)

    if not frames:
        raise RuntimeError(f"all {len(files)} inputs failed")

    import pandas as pd
    out = gpd.GeoDataFrame(pd.concat(frames, ignore_index=True), crs=frames[0].crs)
    assert_output_sane(out)

    dest = Path(cfg["output_dir"]) / "parcels.gpkg"
    dest.parent.mkdir(parents=True, exist_ok=True)
    tmp = dest.with_suffix(".gpkg.part")
    out.to_file(tmp, driver="GPKG")
    tmp.replace(dest)

    return {"features": len(out), "output": str(dest),
            "inputs": len(files), "failures": len(failures),
            "duration_s": round(time.perf_counter() - started, 1)}

def main() -> int:
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)-8s %(message)s")
    try:
        cfg = load_config(sys.argv[1])
    except Exception as exc:
        log.error("config error: %s", exc)
        return 2

    try:
        result = run(cfg)
    except Exception:
        log.exception("run failed")
        write_summary(Path("logs/last_run.json"), status="failed")
        return 1

    write_summary(Path("logs/last_run.json"), status="ok", **result)
    log.info("ok: %(features)s features in %(duration_s)ss", result)
    return 1 if result["failures"] else 0

if __name__ == "__main__":
    raise SystemExit(main())

Example 2: a decorator that guarantees a step is logged 

import functools, logging

log = logging.getLogger("pipeline")

def step(name):
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            log.info("→ %s", name)
            try:
                result = fn(*args, **kwargs)
            except Exception:
                log.exception("✗ %s failed", name)
                raise
            log.info("✓ %s", name)
            return result
        return wrapper
    return decorator

@step("clip to boundary")
def clip_to_boundary(gdf, boundary):
    return gpd.clip(gdf, boundary)

Every step now reports start, success or failure, and no step can fail quietly.

Example 3: guard against a shrinking dataset 

import json
from pathlib import Path

STATE = Path("logs/last_run.json")

def check_against_previous(count: int, tolerance: float = 0.2) -> None:
    if not STATE.exists():
        return
    previous = json.loads(STATE.read_text()).get("features")
    if previous and count < previous * (1 - tolerance):
        raise ValueError(f"feature count dropped {previous} → {count} (>{tolerance:.0%})")

Example 4: fail the shell wrapper properly 

#!/usr/bin/env bash
set -euo pipefail

LOG="/srv/gis/logs/nightly-$(date +%Y%m%d).log"

on_error() {
  local code=$?
  echo "FAILED at line $BASH_LINENO with exit $code" >&2
  curl -fsS -m 10 "https://alerts.internal/hook?job=nightly&status=failed&code=$code" || true
  exit "$code"
}
trap on_error ERR

{
  /srv/gis/.venv/bin/python /srv/gis/nightly.py /srv/gis/configs/daily.yml
} >> "$LOG" 2>&1

curl -fsS -m 10 "https://alerts.internal/hook?job=nightly&status=ok" || true

Explanation

A failure has to travel a long way before a human hears about it. An exception is raised inside a function; a caller either propagates or catches it; main() turns the outcome into an exit code; the process hands that code to a shell; the shell hands its own status to the scheduler; the scheduler decides whether to alert. Each hand-off can drop the signal, and dropping it anywhere makes the whole run look successful.

Checklist of guardrails: no bare except, honest exit code, strict shell, output assertions, structured summary, heartbeat.
Six guardrails, one per hand-off — each closes a different way the signal is lost.

The most common break is the first one. except Exception: pass is usually written to get past one annoying edge case, and then it quietly absorbs every future bug in the same block. The second most common is the exit code: a script that ends by falling off the bottom of the file exits 0, no matter what it logged on the way. The third lives in the shell, where the default is to keep going after an error and report only the last command's status.

Then there is the failure mode that no exception handling can catch, because nothing raised: the job read an empty folder, wrote an empty file, and finished. Correctness here is not about errors at all — it is about expectations. A pipeline that knows how many features it should produce, what CRS the output must carry, and roughly how long it should take can detect wrongness that Python has no opinion about. That is why output assertions belong in the same tier as exception handling rather than in a "nice to have" tier.

Finally, monitoring needs to cover absence. If the scheduler never fired, or the machine was down, there is no log, no exit code and no error to notice — just silence, which reads exactly like a quiet success. A heartbeat or a freshness check on the run summary is the only thing that distinguishes the two.

Edge cases or notes

  • sys.exit("message") exits with code 1: Passing a string prints it to stderr and uses status 1. Pass an int when you mean a specific code.
  • Exceptions in threads do not reach the main thread: A ThreadPoolExecutor stores them in the Future. Call future.result() or the failure is invisible.
  • assert disappears under -O: Never use bare assert for production validation; raise a real exception instead.
  • logging swallows its own errors: A misconfigured handler will not crash the program. Verify the log file is actually being written after a deploy.
  • Warnings are not failures: A GeoSeries.crs warning is easy to lose in a long log. Elevate the ones that matter with warnings.simplefilter("error", UserWarning) in tests.
  • Exit codes above 255 wrap: On POSIX, sys.exit(256) becomes 0. Keep codes between 1 and 125.
  • A finally that returns swallows the exception: Returning from finally discards the in-flight exception entirely. Never do it.

FAQ

Why does my scheduler show a successful run when the job clearly failed?

Because the process exited 0. Either the exception was caught and not re-raised, or main() returned nothing. End the script with raise SystemExit(main()) and return non-zero on failure.

Is except Exception always wrong?

No — it is right at a batch boundary where one item's failure must not stop the others, provided you record the exception. It is wrong when it hides the error, which is why pass is the real problem rather than the broad catch.

How do I make a shell wrapper fail properly?

Start it with set -euo pipefail, and add a trap ... ERR handler if you want to send an alert. Without pipefail, piping output through tee masks the script's exit code entirely.

What should the exit code be when there was nothing to process?

Decide deliberately and document it. Many pipelines treat an empty input as a failure, because "no files today" usually means an upstream delivery is missing rather than genuinely nothing to do.

How do I detect a job that never ran at all?

Write a run summary with a timestamp on every success, and alert when it becomes stale. A heartbeat ping to a cron-monitoring service does the same thing without extra infrastructure.

Should validation raise or just log a warning?

Raise for anything that makes the output unusable — zero rows, missing CRS, invalid geometries. Warn for conditions worth watching that do not invalidate the result, and put those counts in the run summary so a trend is visible.

How do I stop a partially written output from looking like a success?

Write to a temporary path and rename it into place once the write returns. Path.replace() is atomic, so the final file only ever exists in a complete state.