Cron, systemd, Airflow or CI? Choosing a Scheduler for GIS Jobs

Problem statement

The pipeline works. Now something has to run it at 02:30 every day, and the options are not obviously comparable:

cron              one line, no dependencies, no logs, no retries
systemd timer     working directory, env file, journal, missed-run recovery
Task Scheduler    the Windows equivalent, with its own quirks
GitHub Actions    free scheduling, logs, artifacts, secrets — 6-hour limit, no local data
Airflow           dependency graphs, backfills, a UI — a service to run and maintain
Prefect / Dagster the modern equivalents, cloud or self-hosted
Kubernetes CronJob if you already run Kubernetes

Picking wrongly costs in both directions. Choose cron for a workflow with real dependencies and you end up encoding a DAG in sleep statements. Choose Airflow for one nightly script and you have taken on a database, a scheduler process and a web server to run twelve lines of Python.

Quick answer

Choose by what the workload actually needs:

One job, one machine, no dependencies → cron, behind a wrapper script
Same, but you want logs, env files and missed-run recovery → systemd timer
The code is in git and the data is remote → GitHub Actions / GitLab CI schedule
Several jobs with real dependencies, backfills and retries → Airflow, Prefect or Dagster
You already run containers on a cluster → Kubernetes CronJob

# cron — simplest thing that works
30 2 * * * /srv/gis/run.sh >> /srv/gis/logs/pipeline.log 2>&1

# systemd — same schedule, with a working directory, env file and journal logging
[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

The honest default for a single GIS pipeline is a systemd timer where systemd exists, and cron plus a wrapper script where it does not. Most teams reach for an orchestrator well before they need one.

What each option gives you

Grid comparing cron, systemd, CI schedules, Airflow and Kubernetes CronJob across capabilities. — Six schedulers, and the four capabilities that usually decide between them.

Step-by-step solution

Decision tree choosing a scheduler from dependencies, environment and operational needs. — Four questions that settle the choice — dependencies is the one that changes the answer most.

cron: the baseline

# crontab -e
30 2 * * * /srv/gis/run_parcels.sh

#!/usr/bin/env bash
# /srv/gis/run_parcels.sh — everything cron does not give you
set -euo pipefail

cd /srv/gis
set -a; . /srv/gis/job.env; set +a

LOG="/srv/gis/logs/parcels-$(date +%Y%m%d).log"
{
  echo "=== start $(date -Is) ==="
  flock -n /var/lock/parcels.lock \
    /srv/gis/.venv/bin/python -m src.pipeline --config configs/daily.yml
  echo "=== end $(date -Is) exit=$? ==="
} >> "$LOG" 2>&1

Cron is a timer and nothing else: no environment, no working directory, no logs, no retries, no concept of a missed run. The wrapper above supplies each of those — which is exactly the point. Everything cron lacks is a few lines of shell, and for a single job that is often the right trade.

What it genuinely cannot do: recover a run missed while the machine was off, express a dependency between jobs, or tell you whether last night's run succeeded.

systemd timers: cron with the missing pieces

# /etc/systemd/system/parcels.service
[Unit]
Description=Nightly parcels pipeline
After=network-online.target

[Service]
Type=oneshot
User=gis
WorkingDirectory=/srv/gis
EnvironmentFile=/srv/gis/job.env
ExecStart=/srv/gis/.venv/bin/python -m src.pipeline --config configs/daily.yml
TimeoutStartSec=3600
Nice=10
IOSchedulingClass=idle

# /etc/systemd/system/parcels.timer
[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true          # run a job missed while the machine was off
RandomizedDelaySec=300   # avoid a thundering herd across many hosts
AccuracySec=1min

[Install]
WantedBy=timers.target

sudo systemctl enable --now parcels.timer
systemctl list-timers parcels.timer      # when it last ran, when it runs next
journalctl -u parcels.service -n 100     # the logs, already captured
systemctl start parcels.service          # run it now, by hand

The four things this buys over cron are exactly the four things the wrapper script had to fake: WorkingDirectory, EnvironmentFile, journal logging, and Persistent=true. Add systemctl list-timers, which answers "did it run?" without reading a log, and it is the natural choice on any Linux host.

Windows Task Scheduler

$action = New-ScheduledTaskAction `
  -Execute "C:\srv\gis\.venv\Scripts\python.exe" `
  -Argument "-m src.pipeline --config configs\daily.yml" `
  -WorkingDirectory "C:\srv\gis"

$trigger = New-ScheduledTaskTrigger -Daily -At 2:30am

$settings = New-ScheduledTaskSettingsSet `
  -StartWhenAvailable `           # the equivalent of Persistent=true
  -ExecutionTimeLimit (New-TimeSpan -Hours 2) `
  -RestartCount 2 -RestartInterval (New-TimeSpan -Minutes 15)

Register-ScheduledTask -TaskName "GIS parcels nightly" `
  -Action $action -Trigger $trigger -Settings $settings -User "SVC_GIS" -Password $pw

The traps are the familiar ones in Windows dress: set "Start in" or relative paths break, use the venv's python.exe by full path, and remember that "run whether user is logged on or not" removes access to mapped network drives — use UNC paths instead.

CI schedules: the code is already there

# .github/workflows/nightly.yml
name: nightly parcels
on:
  schedule: [{ cron: "30 2 * * *" }]     # UTC
  workflow_dispatch:

jobs:
  run:
    runs-on: ubuntu-latest
    container: ghcr.io/osgeo/gdal:ubuntu-small-3.9.2
    timeout-minutes: 90
    steps:
      - uses: actions/checkout@v4
      - run: pip3 install --break-system-packages -r requirements.txt
      - run: python3 -m src.pipeline --config configs/daily.yml
        env:
          PGHOST: ${{ secrets.PGHOST }}
          PGPASSWORD: ${{ secrets.PGPASSWORD }}
      - uses: actions/upload-artifact@v4
        with: { name: output, path: data/out/**, retention-days: 30 }

For a pipeline whose inputs and outputs live in cloud storage or a database, this is a complete scheduler with logs, secrets, artifacts, alerting on failure and a manual-run button — and no infrastructure to maintain.

Its limits are real: a job time limit (six hours on hosted runners), modest CPU and memory, no access to on-premises data, and — the one that catches people — GitHub disables scheduled workflows in repositories with no activity for 60 days.

Orchestrators: when there is a graph

# Airflow — the point is the dependency graph, not the schedule
from airflow.decorators import dag, task
from datetime import datetime, timedelta

@dag(schedule="30 2 * * *", start_date=datetime(2026, 1, 1), catchup=True,
     default_args={"retries": 2, "retry_delay": timedelta(minutes=10)})
def parcels_pipeline():

    @task
    def extract_parcels(ds=None) -> str: ...

    @task
    def extract_owners(ds=None) -> str: ...

    @task
    def clean(path: str) -> str: ...

    @task
    def join(parcels: str, owners: str) -> str: ...

    @task
    def publish(path: str) -> None: ...

    publish(join(clean(extract_parcels()), extract_owners()))

parcels_pipeline()

What an orchestrator adds over a timer: a dependency graph, per-task retries, backfills over historical dates (catchup=True and the ds parameter), a UI showing which task failed and when, connection and secret management, and parallel task execution.

What it costs: a scheduler process, a metadata database, a web server, upgrades, and a learning curve. Prefect and Dagster make the deployment lighter and the local development story better, but the trade is the same shape.

Kubernetes CronJob: if you are already there

apiVersion: batch/v1
kind: CronJob
metadata:
  name: parcels-nightly
spec:
  schedule: "30 2 * * *"
  timeZone: "Europe/London"
  concurrencyPolicy: Forbid            # never overlap with itself
  startingDeadlineSeconds: 3600
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 2                  # retries
      activeDeadlineSeconds: 5400
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: pipeline
              image: ghcr.io/acme/gis-pipeline:2026.08.11
              args: ["--config", "/configs/daily.yml"]
              envFrom: [{ secretRef: { name: gis-db } }]
              resources:
                requests: { memory: "2Gi", cpu: "500m" }
                limits:   { memory: "8Gi", cpu: "2" }

Concurrency policy, retries, resource limits and time zones are all declarative, which is genuinely nice — but only worth it if a cluster already exists for other reasons.

The choice, made concrete

def recommend_scheduler(*, jobs=1, has_dependencies=False, needs_backfill=False,
                        data_is_local=True, has_kubernetes=False,
                        runtime_hours=1.0, team_size=1) -> str:
    if has_dependencies and (jobs > 3 or needs_backfill):
        return "Airflow / Prefect / Dagster — you have a real DAG"
    if has_kubernetes and not data_is_local:
        return "Kubernetes CronJob — the platform is already there"
    if not data_is_local and runtime_hours < 5:
        return "GitHub Actions schedule — no infrastructure to run"
    if data_is_local:
        return "systemd timer — logs, env, missed-run recovery, no services"
    return "cron with a wrapper script"

print(recommend_scheduler(jobs=1, data_is_local=True))
print(recommend_scheduler(jobs=6, has_dependencies=True, needs_backfill=True))
print(recommend_scheduler(jobs=2, data_is_local=False, runtime_hours=0.5))

Code examples

Example 1: a wrapper that makes cron behave

#!/usr/bin/env bash
# /srv/gis/run.sh — the missing half of cron
set -euo pipefail

JOB="${1:?usage: run.sh <job-name>}"
BASE=/srv/gis
LOG="$BASE/logs/$JOB-$(date +%Y%m%d).log"
LOCK="/var/lock/gis-$JOB.lock"

mkdir -p "$BASE/logs"
cd "$BASE"
set -a; . "$BASE/job.env"; set +a

on_exit() {
  local code=$?
  echo "=== end $(date -Is) exit=$code ===" >> "$LOG"
  if [ "$code" -ne 0 ]; then
    curl -fsS -m 10 -X POST "$ALERT_WEBHOOK_URL" \
      -H 'Content-Type: application/json' \
      -d "{\"text\":\"🔴 $JOB failed (exit $code) — see $LOG\"}" || true
  else
    curl -fsS -m 10 "$HEARTBEAT_URL/$JOB" || true      # dead-man's switch
  fi
}
trap on_exit EXIT

{
  echo "=== start $(date -Is) host=$(hostname) ==="
  flock -n "$LOCK" "$BASE/.venv/bin/python" -m "src.$JOB" --config "configs/$JOB.yml"
} >> "$LOG" 2>&1

flock -n prevents overlapping runs, the trap reports the outcome either way, and the heartbeat ping covers the "never ran" case. That is most of what an orchestrator would have given you.

Example 2: a systemd unit with everything set

# /etc/systemd/system/parcels.service
[Unit]
Description=Parcels pipeline
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=gis
Group=gis
WorkingDirectory=/srv/gis
EnvironmentFile=/srv/gis/job.env
Environment=PYTHONUNBUFFERED=1
ExecStartPre=/usr/bin/test -d /srv/gis/data/raw
ExecStart=/srv/gis/.venv/bin/python -m src.pipeline --config configs/daily.yml
TimeoutStartSec=5400
MemoryMax=8G
Nice=10
StandardOutput=journal
StandardError=journal

systemctl list-timers --all | grep parcels
journalctl -u parcels.service --since "2 days ago" -p warning
systemd-analyze calendar "*-*-* 02:30:00"     # check a schedule expression

MemoryMax is worth setting: it turns a runaway job into a contained failure rather than an unresponsive host.

Example 3: the same pipeline, three schedulers

# src/pipeline.py — scheduler-agnostic by design
import argparse
from datetime import date
from pathlib import Path

def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--config", type=Path, required=True)
    ap.add_argument("--run-date", type=date.fromisoformat, default=date.today())
    args = ap.parse_args()
    return run(args.config, args.run_date)      # returns 0, 1 or 2

if __name__ == "__main__":
    raise SystemExit(main())

# cron
30 2 * * * /srv/gis/run.sh parcels

# systemd
ExecStart=/srv/gis/.venv/bin/python -m src.pipeline --config configs/daily.yml

# Airflow
BashOperator(task_id="parcels",
             bash_command="python -m src.pipeline --config configs/daily.yml "
                          "--run-date {{ ds }}")

Because the pipeline takes its date as an argument and reports its outcome through an exit code, swapping schedulers is a configuration change rather than a rewrite. That is the property to protect — it is what makes the choice reversible.

Example 4: know whether the scheduler is doing its job

#!/usr/bin/env python3
"""Check every scheduled job's last success, whatever runs it."""
import json, subprocess, sys, time
from pathlib import Path

def systemd_last_run(unit: str) -> str | None:
    out = subprocess.run(["systemctl", "show", unit, "--property=ExecMainExitTimestamp"],
                         capture_output=True, text=True)
    return out.stdout.strip().split("=", 1)[-1] or None

def state_file_age_hours(path: Path) -> float | None:
    return (time.time() - path.stat().st_mtime) / 3600 if path.exists() else None

JOBS = [
    {"name": "parcels", "unit": "parcels.service",
     "state": Path("/srv/gis/logs/last_run.json"), "max_age_h": 26},
]

problems = []
for job in JOBS:
    age = state_file_age_hours(job["state"])
    if age is None:
        problems.append(f"{job['name']}: no successful run recorded")
    elif age > job["max_age_h"]:
        problems.append(f"{job['name']}: last success {age:.1f}h ago "
                        f"(systemd says {systemd_last_run(job['unit'])})")

for problem in problems:
    print(problem, file=sys.stderr)
raise SystemExit(1 if problems else 0)

Explanation

Every scheduler answers the same question — "run this at that time" — and they differ in what they do about the four things that surround it: environment, failure, dependencies and visibility.

Bar chart of operational overhead versus capability for each scheduler option. — Capability and overhead rise together — the skill is stopping at the level you need.

Environment is where cron's minimalism costs the most. It runs your command with /bin/sh, a bare PATH, the user's home as the working directory and no profile, which is why "works manually, fails from cron" is such a common report. systemd solves it declaratively with WorkingDirectory and EnvironmentFile; containers solve it by shipping the environment with the code; a wrapper script solves it by hand.

Failure means three separate things: retries, missed runs and notification. Cron has none of them — a run that fails is simply gone, and one missed while the machine was off never happens. Persistent=true on a systemd timer recovers missed runs; orchestrators and Kubernetes add retries with backoff; CI adds failure notification for free. Everything else you build yourself with a trap and a webhook.

Dependencies are the one capability that genuinely justifies an orchestrator. Three jobs where the second needs the first's output can be expressed with careful scheduling and a lot of hope, but as soon as you want "run C when both A and B have succeeded, retry B twice, and backfill the last month", you are describing a DAG — and re-implementing that on cron is a project.

Visibility is where the gap is widest and most underestimated. systemctl list-timers and journalctl answer "did it run and what happened" in one command. Cron answers neither without a wrapper. An orchestrator gives you a UI showing every task's history. CI gives you a log per run with artifacts attached.

Which leads to the practical advice. Start at the lowest level that meets the need, keep the pipeline itself scheduler-agnostic — arguments in, exit code out, no scheduler-specific code — and move up only when a specific capability is missing. The migration is then cheap, and you avoid the very common outcome of maintaining an Airflow deployment to run one nightly script.

Edge cases or notes

Cron is in the machine's local time zone: Daylight saving means a 02:30 job can run twice or not at all. Run servers in UTC.
GitHub Actions cron is UTC and imprecise: Jobs queue at busy times, and scheduled workflows are disabled after 60 days of repository inactivity.
Persistent=true fires immediately after boot: Useful, occasionally surprising. RandomizedDelaySec spreads the load.
Overlapping runs are the classic bug: Use flock -n with cron, concurrencyPolicy: Forbid in Kubernetes, or max_active_runs=1 in Airflow.
Airflow tasks should be small: A single task that runs a four-hour script gets you the overhead without the benefits.
Time zones in orchestrators are explicit: Both Airflow and Kubernetes CronJob accept a timeZone — set it rather than assuming.
The scheduler is not the monitor: Whatever runs the job, something else must check that it ran.

Internal links

FAQ

Is cron good enough for a GIS pipeline?

For one job on one machine, yes — behind a wrapper script that supplies the working directory, environment, logging, locking and alerting. Those five lines are what cron leaves to you.

What does a systemd timer add over cron?

A working directory, an environment file, journal logging, missed-run recovery with Persistent=true, resource limits, and systemctl list-timers to answer "did it run?" without reading a log.

When is Airflow worth it?

When you have a genuine dependency graph, need backfills over historical dates, or run enough jobs that a shared UI and retry policy pay for the scheduler, database and web server you now operate.

Can I use GitHub Actions as a scheduler?

Yes, and it is excellent when the code is in git and the data is in cloud storage or a database. Watch the job time limit, the runner's modest resources, and the 60-day inactivity rule that disables schedules.

How do I stop a job overlapping with itself?

flock -n in a cron wrapper, concurrencyPolicy: Forbid for Kubernetes, max_active_runs=1 in Airflow. systemd's oneshot services will not start a second instance.

What about Prefect or Dagster?

Both offer the orchestrator features with a lighter deployment and a much better local development story than Airflow. If you have decided you need an orchestrator, they are worth evaluating first.

How do I keep the option to change my mind?

Keep the pipeline scheduler-agnostic: parameters in, exit code out, no scheduler-specific imports. Then moving from cron to systemd to an orchestrator is a configuration change, not a rewrite.

Cron, systemd, Airflow or CI? Choosing a Scheduler for GIS Jobs #

Problem statement #

Quick answer #

What each option gives you #

Step-by-step solution #

cron: the baseline #

systemd timers: cron with the missing pieces #

Windows Task Scheduler #

CI schedules: the code is already there #

Orchestrators: when there is a graph #

Kubernetes CronJob: if you are already there #

The choice, made concrete #

Code examples #

Example 1: a wrapper that makes cron behave #

Example 2: a systemd unit with everything set #

Example 3: the same pipeline, three schedulers #

Example 4: know whether the scheduler is doing its job #

Explanation #

Edge cases or notes #

Internal links #

FAQ #

Is cron good enough for a GIS pipeline? #

What does a systemd timer add over cron? #

When is Airflow worth it? #

Can I use GitHub Actions as a scheduler? #

How do I stop a job overlapping with itself? #

What about Prefect or Dagster? #

How do I keep the option to change my mind? #