How to Get Alerted When an Automated GIS Job Fails

On this page

Silence from a scheduler means nothing. It is the exact same signal for "ran perfectly at 02:15 and wrote every feature" and "the virtual environment broke in March and nothing has run since". The layer everyone uses still exists, still opens in QGIS, still looks reasonable — it is simply five months old. Every team that automates GIS work eventually discovers a dead job this way, usually because someone downstream noticed a date. Alerting is how you make silence meaningful: the job tells you when it fails, and — just as importantly — tells you when it has stopped telling you anything at all.

Problem statement

Automated jobs fail without anyone finding out. The concrete gaps:

Failures go to a log nobody reads — the traceback is in /var/log, and no human has opened it since setup.
Cron's mail goes nowhere — the default MAILTO behaviour depends on a mail transfer agent that was never configured.
Success is assumed from silence — nothing distinguishes a healthy quiet night from a machine that rebooted.
Alerts that fire too often are ignored — a job that emails on every warning trains everyone to filter it away.
Alerts with no detail — "Job failed" with no stage, no error, and no log excerpt, at two in the morning.

A failed run should reach a person through a channel they watch, with enough context to act; a run that never happened should be noticed too; and alerts should stay rare enough to mean something.

Quick answer

Wrap the entry point in a try/except that sends a notification and re-raises. Send to whichever channel your team actually reads — email, Slack or Teams webhook, or an alerting service — and include the job name, the stage, the exception, and the last lines of the log.

One failure handler, several delivery channels — and always a non-zero exit code.

def notify(subject: str, body: str, level: str = "error"):
    url = os.environ.get("ALERT_WEBHOOK_URL")
    if not url:
        log.warning("no ALERT_WEBHOOK_URL set; alert not sent: %s", subject)
        return
    icon = {"error": "🔴", "warning": "🟠", "ok": "🟢"}.get(level, "🔵")
    try:
        requests.post(url, json={"text": f"{icon} *{subject}*\n```{body}```"}, timeout=10)
    except Exception:
        log.exception("failed to send alert")      # never let alerting kill the job


def main(cfg=GOOD):
    try:
        result = run_pipeline(cfg)
    except Exception as exc:
        notify(f"{JOB} FAILED on {socket.gethostname()}",
               f"{datetime.now(timezone.utc):%Y-%m-%d %H:%M UTC}\n{type(exc).__name__}: {exc}\n\n"
               + "".join(traceback.format_exc().splitlines(keepends=True)[-6:]))
        raise                     # re-raise so the exit code is non-zero
    return result


print("good run:", main())
try:
    main({**GOOD, "url": "https://api.ratings.food.gov.uk/Establishment"})     # one letter missing
except Exception as exc:
    print("bad run raised:", type(exc).__name__)
print(len(received), "alert(s) delivered:")
print(json.loads(received[-1]["body"])["text"].split("\n")[0].replace(socket.gethostname(), "<host>"))

good run: {'output_rows': 1326, 'input_rows': 1701, 'seconds': 0.6}
bad run raised: StageError
1 alert(s) delivered:
🔴 *food-register-nightly FAILED on <host>*

The good run sends nothing; the bad run sends one message and still raises, so the process exits non-zero. (A sys.exit(0 if main() else 1) at the bottom of a script turns the result into the exit code.)

Decide what deserves an alert, then send it from one place

Decide what deserves an alert

An alert is an interruption, and interruptions have a budget. Spend it on things a human must act on.

Most events belong in the log. Reserve notifications for what needs a decision.

Alert — the job crashed, the output failed validation, the job did not run at all, or the result changed by more than a threshold. Log only — retries that eventually succeeded, individual bad files in a batch, slower-than-usual runtime, warnings from a cleaning stage. Summarise — per-file failures in a large batch, collected into one message at the end rather than one message each.

The failure mode to avoid is alerting on things that are merely unusual. A job that notifies on every warning becomes a filter rule in someone's inbox within a fortnight, and after that the real failure is invisible too.

Send the alert from one place

Failure handling belongs in the entry point, not scattered through stages. One handler means a consistent message format and one place to change the channel. In outline (the functions it calls are written out below):

def main(cfg):
    started = datetime.now(timezone.utc)
    try:
        result = run_pipeline(cfg)
    except Exception as exc:
        alert_failure(JOB, exc, started, cfg)
        raise
    else:
        heartbeat_ok(JOB, result)
        return result

Make the message answer the obvious questions

Read your own alert at 2 a.m. and ask what you would need. Which job, on which machine, when, at which stage, what error, and where is the log.

def read_log_tail(path, lines=15):
    return Path(path).read_text().splitlines()[-lines:]


def alert_failure(job, exc, started, cfg):
    elapsed = (datetime.now(timezone.utc) - started).total_seconds()
    tail = "\n".join(read_log_tail(cfg["log_file"], lines=15))
    body = (
        f"job      : {job}\n"
        f"host     : {socket.gethostname()}\n"
        f"started  : {started:%Y-%m-%d %H:%M:%S UTC} ({elapsed:.0f}s ago)\n"
        f"stage    : {getattr(exc, 'stage', 'unknown')}\n"
        f"error    : {type(exc).__name__}: {exc}\n"
        f"config   : {cfg.get('_config_path')}\n"
        f"log      : {cfg['log_file']}\n\n"
        f"--- last 15 log lines ---\n{tail}"
    )
    notify(f"{job} FAILED", body, level="error")


started = datetime.now(timezone.utc)
try:
    run_pipeline({**GOOD, "url": "https://api.ratings.food.gov.uk/Establishment"})
except StageError as exc:
    alert_failure(JOB, exc, started, GOOD)
text = json.loads(received[-1]["body"])["text"]
print("\n".join(line.strip("`") for line in text.splitlines()[1:8]).replace(socket.gethostname(), "<host>"))

job      : food-register-nightly
host     : <host>
started  : 2026-10-10 22:38:30 UTC (0s ago)
stage    : fetch
error    : StageError: HTTPError: 404 Client Error: Not Found for url: https://api.ratings.food.gov.uk/Establishment?localAuthorityId=284&pageSize=5000
config   : config/prod.yml
log      : logs/pipeline.log

Attaching the stage name is why the pipeline runner wraps stage calls in a StageError that records which one failed — a message saying "failed in join_wards" is worth ten generic tracebacks.

Pick a channel people actually watch

Email works everywhere and is easy to ignore. Good for daily summaries, weak for urgent failures. Chat webhooks (Slack, Teams, Discord) land where the team already is, take five minutes to set up, and are the right default for most GIS teams. Alerting services (PagerDuty, Opsgenie) add on-call rotation and escalation — worth it only when someone is genuinely on call. Dead-man's-switch services (Healthchecks.io, Cronitor) alert on absence, which is the failure mode the others miss entirely.

Most teams need the second row and the fourth, and nothing else.

def send_email(subject, body, cfg):
    import smtplib
    from email.message import EmailMessage
    msg = EmailMessage()
    msg["Subject"], msg["From"], msg["To"] = subject, cfg["from"], ", ".join(cfg["to"])
    msg.set_content(body)
    with smtplib.SMTP(cfg["host"], cfg["port"], timeout=20) as smtp:
        smtp.starttls()
        smtp.login(cfg["user"], os.environ["SMTP_PASSWORD"])
        smtp.send_message(msg)

This one is not run on this page, because it needs a real mail server and account.

Detect the job that never ran

This is the one no failure handler can catch: if the process never starts, it cannot send anything. The answer is a heartbeat — the job checks in on success, and something external notices when a check-in is missing.

HEARTBEAT_URL = os.environ.get("HEARTBEAT_URL")   # e.g. https://hc-ping.com/<uuid>


def heartbeat(status="success", detail=""):
    if not HEARTBEAT_URL:
        return
    url = HEARTBEAT_URL if status == "success" else f"{HEARTBEAT_URL}/fail"
    try:
        requests.post(url, data=detail[:2000], timeout=10)
    except Exception:
        log.warning("heartbeat ping failed (job result unaffected)")


result = run_pipeline()
heartbeat("success", json.dumps(result))
Path("logs/last_run.json").write_text(json.dumps({"at": datetime.now(timezone.utc).isoformat(), **result}))
heartbeat("fail", "fetch: HTTPError 404")
print([r["path"] for r in received[-2:]], "|", received[-2]["body"])

['/ping/food-register-nightly', '/ping/food-register-nightly/fail'] | {"output_rows": 1326, "input_rows": 1701, "seconds": 0.7}

Without an external service, a local check works too: write a timestamp on success and have a second, much simpler scheduled job complain when the file gets old.

# watchdog.py — scheduled hourly; alerts when the nightly job goes quiet
def watchdog(state_file="logs/last_run.json", limit=timedelta(hours=26)):
    state = json.loads(Path(state_file).read_text())
    age = datetime.now(timezone.utc) - datetime.fromisoformat(state["at"])
    if age > limit:
        notify(f"{JOB} has not run", f"last successful run: {state['at']} ({age} ago)")
        return 1
    return 0


print("fresh state ->", watchdog())
stale = json.loads(Path("logs/last_run.json").read_text())
stale["at"] = (datetime.now(timezone.utc) - timedelta(hours=30)).isoformat()   # as if the job stopped
Path("logs/stale_run.json").write_text(json.dumps(stale))
print("30 hours old ->", watchdog("logs/stale_run.json"), "|", json.loads(received[-1]["body"])["text"].split("\n")[0])

fresh state -> 0
30 hours old -> 1 | 🔴 *food-register-nightly has not run*

Twenty-six hours, not twenty-four: a threshold with no slack alerts every time the job runs a few minutes late.

Alert on wrong results, not only on crashes

A job can succeed and still be wrong. Feed the output validation results into the same notifier so a suspicious result is as visible as a crash.

def load_previous_manifest():
    path = Path("logs/last_run.json")
    return json.loads(path.read_text()) if path.exists() else None


def check_and_alert(manifest):
    problems = []
    if manifest["output_rows"] == 0:
        problems.append("output is empty")
    previous = load_previous_manifest()
    if previous:
        change = abs(manifest["output_rows"] - previous["output_rows"]) / max(previous["output_rows"], 1)
        if change > 0.2:
            problems.append(f"row count changed {change:.0%} ({previous['output_rows']} → {manifest['output_rows']})")
        if manifest["seconds"] > 3 * max(previous["seconds"], 1):
            problems.append(f"runtime tripled ({manifest['seconds']}s)")
    if problems:
        notify(f"{JOB} completed with warnings", "\n".join(problems), level="warning")
    return problems


print("same config:", check_and_alert(run_pipeline()))
print("wrong authority id:", check_and_alert(run_pipeline({**GOOD, "authority": 285})))

same config: []
wrong authority id: ['row count changed 27% (1326 → 1679)']

The second run succeeded — no exception, a valid GeoPackage — but a one-digit slip in the authority ID fetched a different council's register. Only the comparison with the previous run notices.

Do not let alerting break the job

An alerting call is a network call, and network calls fail. Wrap every notification in its own try/except with a timeout, so a Slack outage cannot take down a pipeline that was otherwise fine.

class Webhook:
    def __init__(self, name, url):
        self.name, self.url = name, url

    def send(self, subject, body, level, timeout):
        requests.post(self.url, json={"text": f"{subject}\n{body}"}, timeout=timeout).raise_for_status()


dead = socket.socket()
dead.bind(("127.0.0.1", 0))                      # a port nothing listens on: an outage
CHANNELS = [Webhook("chat", f"{BASE}/hook"), Webhook("pager", f"http://127.0.0.1:{dead.getsockname()[1]}/x")]
dead.close()


def notify_all(subject, body, level="error"):
    for channel in CHANNELS:
        try:
            channel.send(subject, body, level, timeout=10)
        except Exception:
            log.exception("alert channel %s failed", channel.name)


before = len(received)
notify_all(f"{JOB} FAILED", "fetch: HTTPError 404")
print(len(received) - before, "of", len(CHANNELS), "channels delivered; the job carried on")
print([line for line in read_log_tail("logs/pipeline.log", 200) if "alert channel" in line][-1].split(" ", 3)[3])

1 of 2 channels delivered; the job carried on
pipeline alert channel pager failed

Alerts actually sent to a local stand-in

The data and the job used on this page

Every example here actually sends its alerts. The setup starts a small local HTTP server that stands in for Slack, Teams or a heartbeat service: it accepts any POST and records it in received, so the page can show exactly what each alert would have delivered. The job being watched is real: it downloads the Food Standards Agency's hygiene register for the Isle of Wight, keeps the businesses that have coordinates, and writes them to a GeoPackage. Its failure is real too — a mistyped API path, which the FSA answers with an HTTP error.

import json
import logging
import os
import socket
import sys
import threading
import time
import traceback
from datetime import datetime, timedelta, timezone
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path

import geopandas as gpd
import pandas as pd
import requests

received = []                                           # every POST the stand-in service gets


class Receiver(BaseHTTPRequestHandler):
    def do_POST(self):
        body = self.rfile.read(int(self.headers.get("Content-Length", 0))).decode()
        received.append({"path": self.path, "body": body})
        self.send_response(200)
        self.end_headers()

    def log_message(self, *args):
        pass


server = HTTPServer(("127.0.0.1", 0), Receiver)
threading.Thread(target=server.serve_forever, daemon=True).start()
BASE = f"http://127.0.0.1:{server.server_port}"
os.environ.update(ALERT_WEBHOOK_URL=f"{BASE}/hook", SLACK_WEBHOOK_URL=f"{BASE}/slack",
                  TEAMS_WEBHOOK_URL=f"{BASE}/teams", HEARTBEAT_URL=f"{BASE}/ping/food-register-nightly")

Path("logs").mkdir(exist_ok=True)
Path("output").mkdir(exist_ok=True)
logging.basicConfig(filename="logs/pipeline.log", level=logging.INFO, force=True,
                    format="%(asctime)s %(levelname)s %(name)s %(message)s")
log = logging.getLogger("pipeline")
JOB = "food-register-nightly"
GOOD = {"url": "https://api.ratings.food.gov.uk/Establishments", "authority": 284,
        "log_file": "logs/pipeline.log", "_config_path": "config/prod.yml"}


class StageError(RuntimeError):
    def __init__(self, stage, exc):
        super().__init__(f"{type(exc).__name__}: {exc}")
        self.stage = stage


def run_pipeline(cfg=GOOD):
    started = time.perf_counter()
    try:
        log.info("fetch: authority %s", cfg["authority"])
        reply = requests.get(cfg["url"], params={"localAuthorityId": cfg["authority"], "pageSize": 5000},
                             headers={"x-api-version": "2"}, timeout=120)
        reply.raise_for_status()
    except Exception as exc:
        raise StageError("fetch", exc) from exc
    rows = pd.DataFrame(reply.json()["establishments"])
    lon = pd.to_numeric(rows["geocode"].map(lambda g: (g or {}).get("longitude")), errors="coerce")
    lat = pd.to_numeric(rows["geocode"].map(lambda g: (g or {}).get("latitude")), errors="coerce")
    keep = lon.notna() & lat.notna()
    shops = gpd.GeoDataFrame(rows.loc[keep, ["FHRSID", "BusinessName", "RatingValue"]],
                             geometry=gpd.points_from_xy(lon[keep], lat[keep]), crs=4326)
    shops.to_file("output/food_businesses.gpkg")
    log.info("write: %d features", len(shops))
    return {"output_rows": len(shops), "input_rows": len(rows), "seconds": round(time.perf_counter() - started, 1)}


print("stand-in alert service listening on 127.0.0.1; job:", JOB)

stand-in alert service listening on 127.0.0.1; job: food-register-nightly

Four failures from this page, each first seen by a different mechanism; the missed run is simulated with a 30-hour-old timestamp.

Example 1: A Slack or Teams webhook notifier

Both platforms accept a simple JSON POST, so one function covers each. Here they post to the stand-in service, which shows the payload each platform would receive:

def slack(subject, body, level="error"):
    colour = {"error": "#ef4444", "warning": "#d97706", "ok": "#14b8a6"}[level]
    requests.post(os.environ["SLACK_WEBHOOK_URL"], timeout=10,
                  json={"attachments": [{"color": colour, "title": subject, "text": f"```{body[:3500]}```"}]},
                  ).raise_for_status()


def teams(subject, body, level="error"):
    requests.post(os.environ["TEAMS_WEBHOOK_URL"], timeout=10,
                  json={"@type": "MessageCard", "@context": "https://schema.org/extensions",
                        "themeColor": {"error": "ef4444", "warning": "d97706"}.get(level, "0ea5e9"),
                        "title": subject, "text": f"<pre>{body[:3500]}</pre>"},
                  ).raise_for_status()


long_trace = "\n".join(f"line {i}" for i in range(2000))                   # about 20 kB
slack(f"{JOB} FAILED", long_trace)
teams(f"{JOB} FAILED", long_trace)
for r in received[-2:]:
    print(r["path"], f"{len(r['body']):,} bytes:", r["body"][:70], "...")

/slack 3,999 bytes: {"attachments": [{"color": "#ef4444", "title": "food-register-nightly  ...
/teams 4,058 bytes: {"@type": "MessageCard", "@context": "https://schema.org/extensions",  ...

Truncate the body — every platform has a payload limit, and an oversized message is rejected rather than delivered. The 20 kB traceback went out as under 4 kB.

Example 2: A logging handler that alerts on ERROR

Routing alerts through the logging module means any log.error() anywhere can raise the alarm, with no extra plumbing. The cooldown decides how noisy it is — and what it counts as "the same" error decides whether the cooldown works at all:

class AlertHandler(logging.Handler):
    def __init__(self, notifier, level=logging.ERROR, cooldown=300, key=None):
        super().__init__(level)
        self.notifier, self.cooldown, self._last = notifier, cooldown, {}
        self.key = key or (lambda record: (record.name, record.levelno, record.getMessage()[:80]))

    def emit(self, record):
        key = self.key(record)
        now = time.time()
        if now - self._last.get(key, 0) < self.cooldown:
            return                                  # suppress duplicates
        self._last[key] = now
        try:
            self.notifier(f"{record.levelname}: {record.name}", self.format(record))
        except Exception:
            pass                                    # never raise from a handler


paths = [f"supplier/parcels_{i:03d}.shp" for i in range(400)]
for label, key in [("key includes the message", None),
                   ("key is the message template", lambda r: (r.name, r.levelno, r.msg))]:
    sent = []
    handler = AlertHandler(lambda subject, body: sent.append(subject), key=key)
    batch = logging.getLogger("batch")
    batch.addHandler(handler)
    for path in paths:
        batch.error("failed to read %s: missing .shx", path)
    batch.removeHandler(handler)
    print(f"{label:30} {len(sent)} alerts for {len(paths)} failures")

key includes the message       400 alerts for 400 failures
key is the message template    1 alerts for 400 failures

A cooldown keyed on the formatted message suppresses nothing when the message contains the file name, because every message is different: 400 failures, 400 alerts. Keying on the message template (record.msg, before the arguments are filled in) groups them into one.

Measured in Example 2: the same 400 failures, 400 alerts or one, depending only on the cooldown key.

Example 3: One digest instead of many alerts

For batch jobs, collect failures and send a single summary — one message you will read beats fifty you will mute. Here the "batch" is the real register: each business is processed, and those without coordinates fail:

register = pd.DataFrame(requests.get(GOOD["url"], params={"localAuthorityId": 284, "pageSize": 5000},
                                     headers={"x-api-version": "2"}, timeout=120).json()["establishments"])


def process(record):
    geocode = record["geocode"] or {}
    if not geocode.get("latitude"):
        raise ValueError("no coordinates")
    return float(geocode["longitude"]), float(geocode["latitude"])


failures = []
for record in register.to_dict("records"):
    try:
        process(record)
    except Exception as exc:
        failures.append((record["BusinessName"][:38], f"{type(exc).__name__}: {exc}"))

if failures:
    lines = "\n".join(f"  {name:40s} {err}" for name, err in failures[:3])
    more = f"\n  … and {len(failures) - 3} more" if len(failures) > 3 else ""
    level = "error" if len(failures) > len(register) * 0.1 else "warning"
    notify(f"{JOB}: {len(failures)}/{len(register)} records failed", lines + more, level=level)
print(json.loads(received[-1]["body"])["text"][:60], "...", f"(level: {level})")

🔴 *food-register-nightly: 375/1701 records failed*
```  Age  ... (level: error)

Escalating severity by proportion is a useful touch: three bad records out of a thousand is a warning; here a fifth of the register has no location, and that is an error.

Example 4: Alerting from the scheduler instead of the script

When the script dies so badly it cannot send anything — a syntax error, a broken interpreter — the scheduler is the only thing left to report it. A wrapper script, run here against a job that cannot even import:

%%writefile run.sh
#!/usr/bin/env bash
# run.sh — the wrapper cron calls
set -uo pipefail
LOG="logs/job-$(date +%F).log"

python3 -c "import no_such_module" >> "$LOG" 2>&1
status=$?

if [ $status -ne 0 ]; then
  curl -sS -X POST "$ALERT_WEBHOOK_URL" -H 'Content-Type: application/json' \
    -d "$(jq -Rn --arg t "🔴 food-register-nightly exited $status
$(tail -n 3 "$LOG")" '{text:$t}')" || true
fi
exit $status

import subprocess

before = len(received)
run = subprocess.run(["bash", "run.sh"], capture_output=True, text=True)
print("wrapper exit code:", run.returncode, "| alerts:", len(received) - before)
print(json.loads(received[-1]["body"])["text"])

wrapper exit code: 1 | alerts: 1
🔴 food-register-nightly exited 1
  File "<string>", line 1, in <module>
    import no_such_module
ModuleNotFoundError: No module named 'no_such_module'

The Python-level handler never ran — Python failed before any of the job's code existed — but the wrapper still reported the exit code and the last lines of the log. This catches the class of failure a Python-level handler structurally cannot.

Example 5: A systemd `OnFailure` unit

On a systemd machine, failure notification can be a property of the service rather than code you maintain.

# /etc/systemd/system/gis-parcels.service
[Unit]
Description=Parcels GIS pipeline
OnFailure=gis-alert@%n.service

[Service]
Type=oneshot
WorkingDirectory=/srv/gis/parcels
ExecStart=/srv/gis/parcels/.venv/bin/python run_pipeline.py --config config/prod.yml

# /etc/systemd/system/gis-alert@.service
[Unit]
Description=Alert that %i failed

[Service]
Type=oneshot
ExecStart=/usr/local/bin/notify-failure.sh %i

Any unit that exits non-zero now triggers the alert, with no per-job code at all. (These unit files are not run on this page; they need a systemd machine with permission to install units.) See scheduling a Python GIS script for the timer that drives it.

A job that fails and a job that never ran

The central insight is that an automated job has two distinct failure modes, and they need different mechanisms. A job that runs and fails can report on itself — a try/except, a non-zero exit, an alert. A job that never runs cannot: the machine was off, the scheduler entry was deleted, the interpreter is broken. Only something external — a heartbeat check, a watchdog, a monitoring service — can detect that. Teams routinely build the first and skip the second, which is precisely why dead jobs go unnoticed for months.

Exit codes are the substrate everything else rests on. A script that catches an exception, sends a nice alert, and then returns 0 has told the scheduler the run succeeded — so systemd will not trigger OnFailure, CI will go green, and any wrapper checking $? will stay quiet. Always re-raise, or return a non-zero code, after alerting.

Alert fatigue is a real failure mode: the muted channel is the same as no alerting at all.

Alert quality is a design problem, not a formatting one. The measure of a good alert is whether the recipient can decide what to do without opening anything else. Job name, host, time, stage, exception, and fifteen lines of log usually clear that bar. A message reading "Pipeline error" does not, and the cost is not just inconvenience — it is the twenty minutes of ssh and log-hunting that happen every time before anyone knows whether it matters.

Deduplication and cooldowns matter more in GIS than in most domains because batch jobs fail per file. A folder of 400 shapefiles with a bad supplier export can generate 400 alerts in seconds — and, as Example 2 showed, a cooldown keyed on the message text lets every one through when the message names the file. That is functionally identical to sending none. Collect, summarise, and escalate by proportion. And keep the alert path itself defensive: wrapped in try/except, with a short timeout, so an outage at the notification service is a logged inconvenience rather than a pipeline failure.

Alerting that has quietly stopped working

Cron will mail the output to a mailbox that, on a container, does not exist.

Alerting on the run that succeeds every time

If a job alerts only on failure and it has not failed in a year, you have no evidence the alerting still works. Test it deliberately: raise an exception behind a --test-alert flag, or have the watchdog verify the channel weekly. Untested alerting is indistinguishable from broken alerting.

Cron's built-in mail is not a plan

Cron mails a job's output to the crontab owner if a mail transfer agent is configured — which on a modern container or cloud VM it usually is not. Tested in a Debian 13 container with cron and no MTA, a job that printed a line and exited with status 3 ran on schedule; there was no sendmail, and /var/mail stayed empty. The output and the exit status went nowhere. Do not rely on it. Send notifications from the job or the wrapper, where you control delivery and can see failures.

Secrets in webhook URLs

A Slack webhook URL is a credential: anyone holding it can post as your integration. Keep it in an environment variable or a secret manager, never in a committed config file, and rotate it if it leaks. The same applies to SMTP passwords and heartbeat URLs.

Alerts containing data

A traceback can include a file path, a connection string, or a sample of attributes. Chat channels and email are broader audiences than a log on a server. Truncate, redact obvious secrets, and think about who is in the channel before including record-level detail.

Time zones in alert timestamps

Stamp alerts in UTC with the offset shown. An alert saying "failed at 02:15" is ambiguous across a team in three countries, and daylight saving makes local timestamps ambiguous even within one.

Distinguish "no data yet" from "failure"

Some pipelines legitimately find nothing — no new incidents, no updated parcels. If an empty result is normal on Sundays, encode that expectation rather than alerting on it, or the Sunday alert will teach everyone to ignore the Monday one.

Internal links

FAQ

How do I get notified when a scheduled GIS script fails?

Wrap the entry point in a try/except that sends a notification and then re-raises so the exit code stays non-zero. Send to a channel your team actually watches — a Slack or Teams webhook is the pragmatic default — and include the job name, host, stage, exception, and the last lines of the log.

How do I detect a job that never ran at all?

Only something outside the job can. Ping a dead-man's-switch service such as Healthchecks.io or Cronitor on every success and let it alert when a ping is missed, or write a timestamp file on success and have a second, simpler scheduled job alert when that file gets stale. A failure handler inside a process that never starts can do nothing.

Should the script alert, or the scheduler?

Both, because they catch different failures. The script produces rich alerts with stage and context, but cannot report a syntax error or a broken interpreter. A wrapper script or a systemd OnFailure unit catches those, at the cost of less detail. The two together cover the whole space.

How do I avoid alert fatigue?

Alert only on what needs a human decision — a crash, failed validation, or a missing run — and log everything else. Deduplicate with a cooldown so a per-file loop cannot send hundreds of messages, summarise batch failures into one digest, and scale severity by proportion. A channel people mute is worse than no channel.

What should a failure alert contain?

Enough to decide without opening anything: job name, host, timestamp in UTC, which stage failed, the exception type and message, the config used, the log path, and the last ten to fifteen log lines. Truncate to fit the platform's payload limit rather than letting a large traceback cause the delivery to be rejected.

Can alerting break my pipeline?

Yes, if you let it. A webhook POST is a network call that can hang or fail. Give it a short timeout, wrap it in its own try/except, and log rather than raise when it fails. A notification service being down should never turn a successful GIS run into a failed one.

Should I alert when the output looks wrong but the job succeeded?

Yes — that is the most valuable alert you can have. Compare the output row count and runtime against the previous run and notify when either changes by more than a threshold. A pipeline that succeeds while producing a fifth of the usual features is a failure that no exception handler will ever see; see validate pipeline inputs and outputs.