Pipelines & automation

A pipeline is a batch job with its decisions pulled out. The stages are ordinary functions; what makes it a pipeline is the configuration above them, the logging and the alerting below, and the fact that you can swap one stage without touching anything else.

The value shows up on the days it fails. A named stage in an exception message tells you where; a per-stage log tells you what it was doing; an alert tells you at all. Without those, a scheduled job that has not run for three weeks looks exactly like a scheduled job that has.

Below: the concepts cover what belongs in config and what belongs in code; the how-to guides build the runner, the scheduling, the retries and the reporting; and the fixes are the failures peculiar to work that runs unattended — cron with no environment, a job that never started, a retry that made things worse.

Learning paths run through this topic too — see the end of this page.