How to Get Alerted When an Automated GIS Job Fails

Silence from a scheduler means nothing. It is the exact same signal for "ran perfectly at 02:15, wrote 18,204 features" and "the virtual environment broke in March and nothing has run since". The layer everyone uses still exists, still opens in QGIS, still looks reasonable — it is simply five months old. Every team that automates GIS work eventually discovers a dead job this way, usually because someone downstream noticed a date. Alerting is how you make silence meaningful: the job tells you when it fails, and — just as importantly — tells you when it has stopped telling you anything at all.

Problem statement

Automated jobs fail without anyone finding out. The concrete gaps:

Failures go to a log nobody reads — the traceback is in /var/log, and no human has opened it since setup.
Cron's mail goes nowhere — the default MAILTO behaviour depends on a mail transfer agent that was never configured.
Success is assumed from silence — nothing distinguishes a healthy quiet night from a machine that rebooted.
Alerts that fire too often are ignored — a job that emails on every warning trains everyone to filter it away.
Alerts with no detail — "Job failed" with no stage, no error, and no log excerpt, at two in the morning.

The goal: a failed run notifies a human through a channel they actually watch, with enough context to act; a run that does not happen at all is also detected; and the volume stays low enough that an alert still means something.

Quick answer

Wrap the entry point in a try/except that sends a notification and re-raises. Send to whichever channel your team actually reads — email, Slack or Teams webhook, or an alerting service — and include the job name, the stage, the exception, and the last lines of the log.

A pipeline failure routed through a notifier to email, a chat webhook, and a non-zero exit code. — One failure handler, several delivery channels — and always a non-zero exit code.

import logging, os, socket, sys, traceback
from datetime import datetime, timezone
import requests

log = logging.getLogger("pipeline")
JOB = "parcels-nightly"

def notify(subject: str, body: str, level: str = "error"):
    url = os.environ.get("ALERT_WEBHOOK_URL")
    if not url:
        log.warning("no ALERT_WEBHOOK_URL set; alert not sent: %s", subject)
        return
    icon = {"error": "🔴", "warning": "🟠", "ok": "🟢"}.get(level, "🔵")
    try:
        requests.post(url, json={"text": f"{icon} *{subject}*\n```{body}```"}, timeout=10)
    except Exception:
        log.exception("failed to send alert")      # never let alerting kill the job

def main():
    try:
        result = run_pipeline()
    except Exception as exc:
        notify(
            f"{JOB} FAILED on {socket.gethostname()}",
            f"{datetime.now(timezone.utc):%Y-%m-%d %H:%M UTC}\n"
            f"{type(exc).__name__}: {exc}\n\n"
            + "".join(traceback.format_exc().splitlines(keepends=True)[-12:]),
        )
        raise                     # re-raise so the exit code is non-zero
    return result

if __name__ == "__main__":
    sys.exit(0 if main() else 1)

Step-by-step solution

Decide what deserves an alert

An alert is an interruption, and interruptions have a budget. Spend it on things a human must act on.

Three severity levels: page a human, notify a channel, or log only, with example events for each. — Most events belong in the log. Reserve notifications for what needs a decision.

Alert — the job crashed, the output failed validation, the job did not run at all, or the result changed by more than a threshold. Log only — retries that eventually succeeded, individual bad files in a batch, slower-than-usual runtime, warnings from a cleaning stage. Summarise — per-file failures in a large batch, collected into one message at the end rather than one message each.

The failure mode to avoid is alerting on things that are merely unusual. A job that notifies on every warning becomes a filter rule in someone's inbox within a fortnight, and after that the real failure is invisible too.

Send the alert from one place

Failure handling belongs in the entry point, not scattered through stages. One handler means a consistent message format and one place to change the channel.

def main(cfg):
    started = datetime.now(timezone.utc)
    try:
        result = run_pipeline(cfg)
    except Exception as exc:
        alert_failure(JOB, exc, started, cfg)
        raise
    else:
        heartbeat_ok(JOB, result)
        return result

Make the message answer the obvious questions

Read your own alert at 2 a.m. and ask what you would need. Which job, on which machine, when, at which stage, what error, and where is the log.

def alert_failure(job, exc, started, cfg):
    elapsed = (datetime.now(timezone.utc) - started).total_seconds()
    tail = "\n".join(read_log_tail(cfg["log_file"], lines=15))
    body = (
        f"job      : {job}\n"
        f"host     : {socket.gethostname()}\n"
        f"started  : {started:%Y-%m-%d %H:%M:%S UTC} ({elapsed:.0f}s ago)\n"
        f"stage    : {getattr(exc, 'stage', 'unknown')}\n"
        f"error    : {type(exc).__name__}: {exc}\n"
        f"config   : {cfg.get('_config_path')}\n"
        f"log      : {cfg['log_file']}\n\n"
        f"--- last 15 log lines ---\n{tail}"
    )
    notify(f"{job} FAILED", body, level="error")

Attaching the stage name is why the pipeline runner wraps stage calls in a StageError that records which one failed — a message saying "failed in join_wards" is worth ten generic tracebacks.

Pick a channel people actually watch

Email works everywhere and is easy to ignore. Good for daily summaries, weak for urgent failures. Chat webhooks (Slack, Teams, Discord) land where the team already is, take five minutes to set up, and are the right default for most GIS teams. Alerting services (PagerDuty, Opsgenie) add on-call rotation and escalation — worth it only when someone is genuinely on call. Dead-man's-switch services (Healthchecks.io, Cronitor) alert on absence, which is the failure mode the others miss entirely.

def send_email(subject, body, cfg):
    import smtplib
    from email.message import EmailMessage
    msg = EmailMessage()
    msg["Subject"], msg["From"], msg["To"] = subject, cfg["from"], ", ".join(cfg["to"])
    msg.set_content(body)
    with smtplib.SMTP(cfg["host"], cfg["port"], timeout=20) as smtp:
        smtp.starttls()
        smtp.login(cfg["user"], os.environ["SMTP_PASSWORD"])
        smtp.send_message(msg)

Detect the job that never ran

This is the one no failure handler can catch: if the process never starts, it cannot send anything. The answer is a heartbeat — the job checks in on success, and something external notices when a check-in is missing.

import requests

HEARTBEAT_URL = os.environ.get("HEARTBEAT_URL")   # e.g. https://hc-ping.com/<uuid>

def heartbeat(status="success", detail=""):
    if not HEARTBEAT_URL:
        return
    url = HEARTBEAT_URL if status == "success" else f"{HEARTBEAT_URL}/fail"
    try:
        requests.post(url, data=detail[:2000], timeout=10)
    except Exception:
        log.warning("heartbeat ping failed (job result unaffected)")

Without an external service, a local check works too: write a timestamp on success and have a second, much simpler scheduled job complain when the file gets old.

# watchdog.py — scheduled hourly; alerts when the nightly job goes quiet
from datetime import datetime, timezone, timedelta
import json, sys
from pathlib import Path

state = json.loads(Path("logs/last_run.json").read_text())
age = datetime.now(timezone.utc) - datetime.fromisoformat(state["at"])
if age > timedelta(hours=26):
    notify("parcels-nightly has not run", f"last successful run: {state['at']} ({age} ago)")
    sys.exit(1)

Twenty-six hours, not twenty-four: a threshold with no slack alerts every time the job runs a few minutes late.

Alert on wrong results, not only on crashes

A job can succeed and still be wrong. Feed the output validation results into the same notifier so a suspicious result is as visible as a crash.

def check_and_alert(manifest, cfg):
    problems = []
    if manifest["output_rows"] == 0:
        problems.append("output is empty")
    previous = load_previous_manifest()
    if previous:
        change = abs(manifest["output_rows"] - previous["output_rows"]) / max(previous["output_rows"], 1)
        if change > 0.2:
            problems.append(
                f"row count changed {change:.0%} "
                f"({previous['output_rows']} → {manifest['output_rows']})")
    if manifest["seconds"] > 3 * previous.get("seconds", manifest["seconds"]):
        problems.append(f"runtime tripled ({manifest['seconds']}s)")

    if problems:
        notify(f"{JOB} completed with warnings", "\n".join(problems), level="warning")
    return problems

Do not let alerting break the job

An alerting call is a network call, and network calls fail. Wrap every notification in its own try/except with a timeout, so a Slack outage cannot take down a pipeline that was otherwise fine.

def notify(subject, body, level="error"):
    for channel in CHANNELS:
        try:
            channel.send(subject, body, level, timeout=10)
        except Exception:
            log.exception("alert channel %s failed", channel.name)

Code examples

Example 1: A Slack or Teams webhook notifier

Both platforms accept a simple JSON POST, so one function covers each.

import os, requests

def slack(subject, body, level="error"):
    colour = {"error": "#ef4444", "warning": "#d97706", "ok": "#14b8a6"}[level]
    requests.post(
        os.environ["SLACK_WEBHOOK_URL"],
        json={"attachments": [{"color": colour, "title": subject,
                               "text": f"```{body[:3500]}```"}]},
        timeout=10,
    ).raise_for_status()

def teams(subject, body, level="error"):
    requests.post(
        os.environ["TEAMS_WEBHOOK_URL"],
        json={"@type": "MessageCard", "@context": "https://schema.org/extensions",
              "themeColor": {"error": "ef4444", "warning": "d97706"}.get(level, "0ea5e9"),
              "title": subject, "text": f"<pre>{body[:3500]}</pre>"},
        timeout=10,
    ).raise_for_status()

Truncate the body — every platform has a payload limit, and a 2 MB traceback will be rejected rather than delivered.

Example 2: A logging handler that alerts on ERROR

Routing alerts through the logging module means any log.error() anywhere can raise the alarm, with no extra plumbing.

import logging

class AlertHandler(logging.Handler):
    def __init__(self, notifier, level=logging.ERROR, cooldown=300):
        super().__init__(level)
        self.notifier, self.cooldown, self._last = notifier, cooldown, {}

    def emit(self, record):
        import time
        key = (record.name, record.levelno, record.getMessage()[:80])
        now = time.time()
        if now - self._last.get(key, 0) < self.cooldown:
            return                                  # suppress duplicates
        self._last[key] = now
        try:
            self.notifier(f"{record.levelname}: {record.name}", self.format(record))
        except Exception:
            pass                                    # never raise from a handler

logging.getLogger().addHandler(AlertHandler(slack))

The cooldown is essential: a loop that logs an error per file would otherwise send four hundred messages.

Example 3: One digest instead of many alerts

For batch jobs, collect failures and send a single summary — one message you will read beats fifty you will mute.

failures = []
for path in files:
    try:
        process(path)
    except Exception as exc:
        log.exception("failed: %s", path.name)
        failures.append((path.name, f"{type(exc).__name__}: {exc}"))

if failures:
    lines = "\n".join(f"  {name:40s} {err}" for name, err in failures[:25])
    more = f"\n  … and {len(failures) - 25} more" if len(failures) > 25 else ""
    notify(f"{JOB}: {len(failures)}/{len(files)} files failed",
           lines + more,
           level="error" if len(failures) > len(files) * 0.1 else "warning")

Escalating severity by proportion is a useful touch: three bad files out of a thousand is a warning; three hundred is an emergency.

Example 4: Alerting from the scheduler instead of the script

When the script dies so badly it cannot send anything — a syntax error, a broken interpreter — the scheduler is the only thing left to report it.

#!/usr/bin/env bash
# run.sh — the wrapper cron calls
set -uo pipefail
PROJECT=/srv/gis/parcels
LOG="$PROJECT/logs/parcels-$(date +%F).log"

cd "$PROJECT"
.venv/bin/python run_pipeline.py --config config/prod.yml >> "$LOG" 2>&1
status=$?

if [ $status -ne 0 ]; then
  curl -sS -X POST "$ALERT_WEBHOOK_URL" -H 'Content-Type: application/json' \
    -d "$(jq -Rn --arg t "🔴 parcels-nightly exited $status on $(hostname)
$(tail -n 20 "$LOG")" '{text:$t}')" || true
fi
exit $status

This wrapper catches the class of failure a Python-level handler structurally cannot.

Example 5: A systemd `OnFailure` unit

On a systemd machine, failure notification can be a property of the service rather than code you maintain.

# /etc/systemd/system/gis-parcels.service
[Unit]
Description=Parcels GIS pipeline
OnFailure=gis-alert@%n.service

[Service]
Type=oneshot
WorkingDirectory=/srv/gis/parcels
ExecStart=/srv/gis/parcels/.venv/bin/python run_pipeline.py --config config/prod.yml

# /etc/systemd/system/[email protected]
[Unit]
Description=Alert that %i failed

[Service]
Type=oneshot
ExecStart=/usr/local/bin/notify-failure.sh %i

Any unit that exits non-zero now triggers the alert, with no per-job code at all. See scheduling a Python GIS script for the timer that drives it.

Explanation

The central insight is that an automated job has two distinct failure modes, and they need different mechanisms. A job that runs and fails can report on itself — a try/except, a non-zero exit, an alert. A job that never runs cannot: the machine was off, the scheduler entry was deleted, the interpreter is broken. Only something external — a heartbeat check, a watchdog, a monitoring service — can detect that. Teams routinely build the first and skip the second, which is precisely why dead jobs go unnoticed for months.

Exit codes are the substrate everything else rests on. A script that catches an exception, sends a nice alert, and then returns 0 has told the scheduler the run succeeded — so systemd will not trigger OnFailure, CI will go green, and any wrapper checking $? will stay quiet. Always re-raise, or return a non-zero code, after alerting.

A noisy alert stream where every warning notifies, versus a quiet stream where only actionable failures do. — Alert fatigue is a real failure mode: the muted channel is the same as no alerting at all.

Alert quality is a design problem, not a formatting one. The measure of a good alert is whether the recipient can decide what to do without opening anything else. Job name, host, time, stage, exception, and fifteen lines of log usually clear that bar. A message reading "Pipeline error" does not, and the cost is not just inconvenience — it is the twenty minutes of ssh and log-hunting that happen every time before anyone knows whether it matters.

Deduplication and cooldowns matter more in GIS than in most domains because batch jobs fail per file. A folder of 400 shapefiles with a bad supplier export can generate 400 identical alerts in ninety seconds, which is functionally identical to sending none. Collect, summarise, and escalate by proportion. And keep the alert path itself defensive: wrapped in try/except, with a short timeout, so an outage at the notification service is a logged inconvenience rather than a pipeline failure.

Edge cases or notes

Alerting on the run that succeeds every time

If a job alerts only on failure and it has not failed in a year, you have no evidence the alerting still works. Test it deliberately: raise an exception behind a --test-alert flag, or have the watchdog verify the channel weekly. Untested alerting is indistinguishable from broken alerting.

Cron's built-in mail is not a plan

Cron mails a job's output to the crontab owner if a mail transfer agent is configured — which on a modern container or cloud VM it usually is not, so the mail is silently discarded. Do not rely on it. Send notifications from the job or the wrapper, where you control delivery and can see failures.

Secrets in webhook URLs

A Slack webhook URL is a credential: anyone holding it can post as your integration. Keep it in an environment variable or a secret manager, never in a committed config file, and rotate it if it leaks. The same applies to SMTP passwords and heartbeat URLs.

Alerts containing data

A traceback can include a file path, a connection string, or a sample of attributes. Chat channels and email are broader audiences than a log on a server. Truncate, redact obvious secrets, and think about who is in the channel before including record-level detail.

Time zones in alert timestamps

Stamp alerts in UTC with the offset shown. An alert saying "failed at 02:15" is ambiguous across a team in three countries, and daylight saving makes local timestamps ambiguous even within one.

Distinguish "no data yet" from "failure"

Some pipelines legitimately find nothing — no new incidents, no updated parcels. If an empty result is normal on Sundays, encode that expectation rather than alerting on it, or the Sunday alert will teach everyone to ignore the Monday one.

Internal links

FAQ

How do I get notified when a scheduled GIS script fails?

Wrap the entry point in a try/except that sends a notification and then re-raises so the exit code stays non-zero. Send to a channel your team actually watches — a Slack or Teams webhook is the pragmatic default — and include the job name, host, stage, exception, and the last lines of the log.

How do I detect a job that never ran at all?

Only something outside the job can. Ping a dead-man's-switch service such as Healthchecks.io or Cronitor on every success and let it alert when a ping is missed, or write a timestamp file on success and have a second, simpler scheduled job alert when that file gets stale. A failure handler inside a process that never starts can do nothing.

Should the script alert, or the scheduler?

Both, because they catch different failures. The script produces rich alerts with stage and context, but cannot report a syntax error or a broken interpreter. A wrapper script or a systemd OnFailure unit catches those, at the cost of less detail. The two together cover the whole space.

How do I avoid alert fatigue?

Alert only on what needs a human decision — a crash, failed validation, or a missing run — and log everything else. Deduplicate with a cooldown so a per-file loop cannot send hundreds of messages, summarise batch failures into one digest, and scale severity by proportion. A channel people mute is worse than no channel.

What should a failure alert contain?

Enough to decide without opening anything: job name, host, timestamp in UTC, which stage failed, the exception type and message, the config used, the log path, and the last ten to fifteen log lines. Truncate to fit the platform's payload limit rather than letting a large traceback cause the delivery to be rejected.

Can alerting break my pipeline?

Yes, if you let it. A webhook POST is a network call that can hang or fail. Give it a short timeout, wrap it in its own try/except, and log rather than raise when it fails. A notification service being down should never turn a successful GIS run into a failed one.

Should I alert when the output looks wrong but the job succeeded?

Yes — that is the most valuable alert you can have. Compare the output row count and runtime against the previous run and notify when either changes by more than a threshold. A pipeline that succeeds while producing a fifth of the usual features is a failure that no exception handler will ever see; see validate pipeline inputs and outputs.

How to Get Alerted When an Automated GIS Job Fails #

Problem statement #

Quick answer #

Step-by-step solution #

Decide what deserves an alert #

Send the alert from one place #

Make the message answer the obvious questions #

Pick a channel people actually watch #

Detect the job that never ran #

Alert on wrong results, not only on crashes #

Do not let alerting break the job #

Code examples #

Example 1: A Slack or Teams webhook notifier #

Example 2: A logging handler that alerts on ERROR #

Example 3: One digest instead of many alerts #

Example 4: Alerting from the scheduler instead of the script #

Example 5: A systemd OnFailure unit #

Explanation #

Edge cases or notes #

Alerting on the run that succeeds every time #

Cron's built-in mail is not a plan #

Secrets in webhook URLs #

Alerts containing data #

Time zones in alert timestamps #

Distinguish "no data yet" from "failure" #

Internal links #

FAQ #

How do I get notified when a scheduled GIS script fails? #

How do I detect a job that never ran at all? #

Should the script alert, or the scheduler? #

How do I avoid alert fatigue? #

What should a failure alert contain? #

Can alerting break my pipeline? #

Should I alert when the output looks wrong but the job succeeded? #