How to Drive a GIS Pipeline from a YAML Config File in Python

The clearest sign that a script has outgrown itself is a block of constants at the top that you edit before every run. Change the input folder, change the CRS, comment out the clip, change it back. Each edit is a chance to break something, and none of it is recorded anywhere. Moving those settings into a YAML file changes the economics completely: the code stops changing, the config becomes the thing you version, and running last month's job again is a matter of pointing at last month's file.

The pipeline on this page is real and small: HM Land Registry's INSPIRE parcels for Rutland, cleaned, reprojected, clipped to the town of Oakham, measured, and filtered by area — every path and number of it in YAML.

Problem statement

Your pipeline works, but every run needs a code edit. This page moves a real one — 21,253 Rutland land parcels cleaned, reprojected, clipped to Oakham and filtered by area — into YAML: the paths, the parameters and the ordered list of steps. Afterwards one unchanged script ran the default config (4,604 parcels kept) and a three-line production overlay (4,957 kept), refused to overwrite its own output, and saved the merged settings beside the result.

What the constants were costing you:

• Editing code to change a path — the difference between the January and February run is two string literals buried in a script.
• No record of what a run used — the settings that produced an output are gone the moment you edit them for the next run.
• Colleagues cannot run it — their data lives elsewhere, so they must understand your Python before they can change a folder.
• Dev and production drift — you keep two copies of the script with slightly different constants, and they diverge.
• Secrets in the repository — a database password sitting in a committed .py file.

A config file per scenario fixes that: validated before the run starts, the code identical everywhere, and each run's exact settings saved next to its output.

Quick answer

Put paths, parameters, and the step list in a YAML file. Load it with yaml.safe_load(), validate the keys and turn path strings into Path objects immediately, and pass the resulting dictionary into the pipeline you already have.

print(Path("config/parcels.yml").read_text())

# config/parcels.yml
input:
  source: ../data/raw/parcels.gpkg
  boundary: ../data/raw/oakham.geojson
output:
  dir: ../data/processed
  driver: GPKG
params:
  target_crs: EPSG:27700
  min_area_m2: 50
steps: [drop_empty, make_valid, reproject, clip, area, filter_area]

REQUIRED = {"input", "output", "params", "steps"}


def load_config(path):
    path = Path(path)
    cfg = yaml.safe_load(path.read_text())
    missing = REQUIRED - cfg.keys()
    if missing:
        raise ValueError(f"config is missing sections: {sorted(missing)}")
    cfg["input"] = {k: path.parent / v for k, v in cfg["input"].items()}
    cfg["output"]["dir"] = path.parent / cfg["output"]["dir"]
    for name, p in cfg["input"].items():
        if not p.exists():
            raise FileNotFoundError(f"input '{name}' not found: {p}")
    return cfg


cfg = load_config("config/parcels.yml")
print({k: str(v) for k, v in cfg["input"].items()}, cfg["params"], cfg["steps"])

{'source': 'config/../data/raw/parcels.gpkg', 'boundary': 'config/../data/raw/oakham.geojson'} {'target_crs': 'EPSG:27700', 'min_area_m2': 50} ['drop_empty', 'make_valid', 'reproject', 'clip', 'area', 'filter_area']

python run_pipeline.py config/parcels.yml then runs the January job — Example 5 does exactly that — and a second file runs February's. The code never changes.

Load safely, validate, resolve paths, build the steps

Choose YAML, and load it safely

YAML wins for config that humans edit: comments, no trailing-comma traps, readable nesting, and native types for numbers and booleans. JSON is fine but cannot carry comments, and TOML is excellent but awkward for deeply nested structures.

Always safe_load, never load. Plain yaml.load() with an unsafe loader can construct arbitrary Python objects from a file, which is a real risk the moment a config comes from anywhere but your own hands:

tagged = "steps: !!python/object/apply:os.getcwd []"
try:
    yaml.safe_load(tagged)
except yaml.YAMLError as exc:
    print("safe_load:  ", type(exc).__name__, "-", exc.problem)
print("unsafe_load:", yaml.unsafe_load(tagged)["steps"] == os.getcwd(), "— it ran os.getcwd()")

safe_load:   ConstructorError - could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:os.getcwd'
unsafe_load: True — it ran os.getcwd()

safe_load gives you dictionaries, lists, strings, numbers, and booleans — everything a config should contain — and refuses the tag that told the unsafe loader to call a function.

Give the file a shape and keep to it

An unstructured config becomes a junk drawer. Four top-level sections cover almost every pipeline, and the grouping tells a reader what each key is for: input (where data comes from), output (where results go), params (the numbers and codes that tune the run) and steps (which transforms to apply, in order).

Validate before you read a single feature

A config error should stop the run in the first second, not after a long reprojection. Check required sections, check that inputs exist, and check that values are the type you expect.

def validate(cfg):
    problems = []
    for section in ("input", "output", "params", "steps"):
        if section not in cfg:
            problems.append(f"missing section: {section}")
    if problems:
        raise ValueError("; ".join(problems))
    for name, path in cfg["input"].items():
        if not Path(path).exists():
            problems.append(f"input '{name}' does not exist: {Path(path).name}")
    crs = cfg["params"].get("target_crs", "")
    if not str(crs).upper().startswith("EPSG:"):
        problems.append(f"target_crs should look like 'EPSG:27700', got {crs!r}")
    if not isinstance(cfg["steps"], list) or not cfg["steps"]:
        problems.append("steps must be a non-empty list")
    if problems:
        raise ValueError("Invalid config:\n  - " + "\n  - ".join(problems))
    return cfg


broken = yaml.safe_load(Path("config/parcels.yml").read_text()
                        .replace("target_crs: EPSG:27700", "target_crs: 27700")
                        .replace("oakham.geojson", "oakam.geojson")
                        .replace("steps: [", "steps_: ["))
broken["steps"] = "reproject"
broken["input"] = {k: Path("config") / v for k, v in broken["input"].items()}
try:
    validate(broken)
except ValueError as exc:
    print(exc)

Invalid config:
  - input 'boundary' does not exist: oakam.geojson
  - target_crs should look like 'EPSG:27700', got 27700
  - steps must be a non-empty list

Collecting all the problems and reporting them together is worth the extra few lines: three mistakes, one run. The unquoted 27700 is also the one YAML typing trap that matters most here — it parses as an integer, not a CRS string.

Resolve paths relative to the config file

A config that only works from one working directory is a config that will fail under a scheduler. Resolve every relative path against the location of the config file itself, so the same file works from anywhere:

start = Path.cwd()
os.chdir("/tmp")
try:
    naive = Path(yaml.safe_load(Path(start / "config/parcels.yml").read_text())["input"]["source"])
    print("relative to the working directory:", naive.exists())
    print("relative to the config file:      ", ((start / "config") / naive).exists())
finally:
    os.chdir(start)

relative to the working directory: False
relative to the config file:       True

This single detail removes the most common cause of "it works when I run it, but the cron job fails" — see scheduling a Python GIS script.

Turn the step list into functions

If your steps live in a registry, the steps: list in YAML becomes a runnable pipeline — and an unknown step name fails immediately with a helpful message.

%%writefile steps.py
"""The step registry: each step takes a GeoDataFrame and the params, and returns a GeoDataFrame."""
import geopandas as gpd

REGISTRY = {}


def step(fn):
    REGISTRY[fn.__name__] = fn
    return fn


@step
def drop_empty(gdf, params):
    return gdf[gdf.geometry.notna() & ~gdf.geometry.is_empty]


@step
def make_valid(gdf, params):
    return gdf.assign(geometry=gdf.geometry.make_valid())


@step
def reproject(gdf, params):
    return gdf.to_crs(params["target_crs"])


@step
def clip(gdf, params):
    boundary = gpd.read_file(params["boundary"]).to_crs(gdf.crs)
    return gpd.clip(gdf, boundary)


@step
def area(gdf, params):
    return gdf.assign(area_m2=gdf.area.round(1))


@step
def filter_area(gdf, params):
    return gdf[gdf["area_m2"] >= params["min_area_m2"]]


def apply_all(gdf, steps, params, log=print):
    for fn in steps:
        before = len(gdf)
        gdf = fn(gdf, params)
        log(f"{fn.__name__:12} {before:>6} -> {len(gdf):>6}")
    return gdf

from steps import REGISTRY


def build_pipeline(names):
    unknown = [n for n in names if n not in REGISTRY]
    if unknown:
        raise KeyError(f"unknown steps {unknown}; known steps: {sorted(REGISTRY)}")
    return [REGISTRY[n] for n in names]


print([fn.__name__ for fn in build_pipeline(cfg["steps"])])
try:
    build_pipeline(["drop_empty", "reprojet"])
except KeyError as exc:
    print(exc)

['drop_empty', 'make_valid', 'reproject', 'clip', 'area', 'filter_area']
"unknown steps ['reprojet']; known steps: ['area', 'clip', 'drop_empty', 'filter_area', 'make_valid', 'reproject']"

Layer configs instead of duplicating them

Two environments should not mean two copies of every setting. Keep a base.yml with everything shared and small overlay files with only the differences, then merge.

def deep_merge(base, override):
    out = copy.deepcopy(base)
    for key, value in (override or {}).items():
        if isinstance(value, dict) and isinstance(out.get(key), dict):
            out[key] = deep_merge(out[key], value)
        else:
            out[key] = value
    return out


base = yaml.safe_load(Path("config/base.yml").read_text())
prod = yaml.safe_load(Path("config/prod.yml").read_text())
merged = deep_merge(base, prod)
print("prod.yml sets:", prod)
print("merged output:", merged["output"], "| params:", merged["params"])

prod.yml sets: {'output': {'dir': '../data/processed/prod', 'overwrite': True}, 'params': {'min_area_m2': 5}}
merged output: {'dir': '../data/processed/prod', 'driver': 'GPKG', 'overwrite': True} | params: {'target_crs': 'EPSG:27700', 'min_area_m2': 5}

The override file is three lines; everything else comes from the base, and a nested section is merged key by key rather than replaced wholesale.

Save the config with the output

The final habit that makes configs pay off: copy the merged config into the run's output folder. Months later, the folder answers its own questions about what produced it.

def snapshot(cfg, run_dir):
    run_dir.mkdir(parents=True, exist_ok=True)
    serialisable = json.loads(json.dumps(cfg, default=str))   # Paths -> strings
    (run_dir / "config_used.json").write_text(json.dumps(serialisable, indent=2))
    return run_dir / "config_used.json"


print(snapshot(merged, Path("data/processed/prod-run")).read_text()[:160], "…")

{
  "input": {
    "source": "../data/raw/parcels.gpkg",
    "boundary": "../data/raw/oakham.geojson"
  },
  "output": {
    "dir": "../data/processed/prod",
   …

A config-driven run over Oakham's parcels

The data used on this page

HM Land Registry's INSPIRE Index Polygons for Rutland (© Crown copyright and database rights; OGL) and Oakham's three wards from the ONS December 2024 boundaries (OGL). The setup writes the raw inputs and the three config files.

import copy
import json
import os
import re
import sys
from pathlib import Path

import geopandas as gpd
import requests
import yaml

for folder in ("data/raw", "config", "logs"):
    Path(folder).mkdir(parents=True, exist_ok=True)
if not Path("data/Rutland.zip").exists():
    Path("data/Rutland.zip").write_bytes(requests.get(
        "https://use-land-property-data.service.gov.uk/datasets/inspire/download/Rutland.zip", timeout=300).content)
if not Path("data/wards.geojson").exists():
    Path("data/wards.geojson").write_bytes(requests.get(
        "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/"
        "Wards_December_2024_Boundaries_UK_BFC/FeatureServer/0/query",
        params={"where": "LAD24CD='E06000017'", "outFields": "WD24NM", "outSR": 27700, "f": "geojson"},
        timeout=120).content)
gpd.read_file("/vsizip/data/Rutland.zip/Land_Registry_Cadastral_Parcels.gml", columns=["INSPIREID"],
              DOWNLOAD_SCHEMA="NO").to_file("data/raw/parcels.gpkg")
wards = gpd.read_file("data/wards.geojson")
wards[wards["WD24NM"].str.startswith("Oakham")].dissolve().to_crs(4326).to_file("data/raw/oakham.geojson")

Path("config/parcels.yml").write_text("""\
# config/parcels.yml
input:
  source: ../data/raw/parcels.gpkg
  boundary: ../data/raw/oakham.geojson
output:
  dir: ../data/processed
  driver: GPKG
params:
  target_crs: EPSG:27700
  min_area_m2: 50
steps: [drop_empty, make_valid, reproject, clip, area, filter_area]
""")
Path("config/base.yml").write_text(Path("config/parcels.yml").read_text().replace("# config/parcels.yml", "# config/base.yml"))
Path("config/prod.yml").write_text("""\
# config/prod.yml — only what differs in production
output: {dir: ../data/processed/prod, overwrite: true}
params: {min_area_m2: 5}
""")
print(sorted(p.name for p in Path("config").iterdir()), "| inputs:", sorted(p.name for p in Path("data/raw").iterdir()))

['base.yml', 'parcels.yml', 'prod.yml'] | inputs: ['oakham.geojson', 'parcels.gpkg']

Example 1: A complete config file

Every setting the pipeline needs, grouped, commented, and readable by someone who does not know Python.

Path("config/parcels-prod.yml").write_text("""\
# config/parcels-prod.yml
name: parcels-monthly

input:
  source: ../data/raw/parcels.gpkg
  boundary: ../data/raw/oakham.geojson

output:
  dir: ../data/processed
  driver: GPKG
  overwrite: false

params:
  target_crs: EPSG:27700     # British National Grid, metres
  min_area_m2: 50            # drop slivers below this
  simplify_m: 0              # 0 disables simplification

steps:
  - drop_empty
  - make_valid
  - reproject
  - clip
  - area
  - filter_area

logging:
  level: INFO
  file: ../logs/parcels.log
""")
print(yaml.safe_load(Path("config/parcels-prod.yml").read_text())["params"])

{'target_crs': 'EPSG:27700', 'min_area_m2': 50, 'simplify_m': 0}

Example 2: Loader with defaults, merging, and validation

Defaults in code mean a short config file stays valid when you add a new setting. The loader goes in its own module, which the entry point in Example 5 imports:

%%writefile config.py
import copy
from pathlib import Path

import yaml

DEFAULTS = {
    "output": {"driver": "GPKG", "overwrite": False},
    "params": {"min_area_m2": 0, "simplify_m": 0},
    "logging": {"level": "INFO", "file": None},
}


def deep_merge(base, override):
    out = copy.deepcopy(base)
    for k, v in (override or {}).items():
        out[k] = deep_merge(out[k], v) if isinstance(v, dict) and isinstance(out.get(k), dict) else v
    return out


def validate(cfg):
    problems = [f"missing section: {s}" for s in ("input", "output", "params", "steps") if s not in cfg]
    problems += [f"input '{n}' does not exist: {p.name}" for n, p in cfg.get("input", {}).items() if not p.exists()]
    if not str(cfg.get("params", {}).get("target_crs", "")).upper().startswith("EPSG:"):
        problems.append("target_crs should look like 'EPSG:27700'")
    if problems:
        raise ValueError("Invalid config:\n  - " + "\n  - ".join(problems))
    return cfg


def load_config(*paths):
    """Load one or more YAML files, later ones overriding earlier, onto the defaults."""
    cfg = DEFAULTS
    for path in map(Path, paths):
        cfg = deep_merge(cfg, yaml.safe_load(path.read_text()) or {})
    base = Path(paths[0]).resolve().parent
    cfg["input"] = {k: (base / v).resolve() for k, v in cfg["input"].items()}
    cfg["output"]["dir"] = (base / cfg["output"]["dir"]).resolve()
    cfg["params"]["boundary"] = cfg["input"].get("boundary")
    cfg["_config_path"] = " + ".join(Path(p).name for p in paths)
    return validate(cfg)

import config

short = config.load_config("config/parcels.yml")
print("defaults filled in:", short["output"]["overwrite"], short["params"]["simplify_m"], short["logging"])
layered = config.load_config("config/base.yml", "config/prod.yml")
print("layered:", layered["_config_path"], "| min_area_m2:", layered["params"]["min_area_m2"],
      "| overwrite:", layered["output"]["overwrite"])

defaults filled in: False 0 {'level': 'INFO', 'file': None}
layered: base.yml + prod.yml | min_area_m2: 5 | overwrite: True

Example 3: Keeping secrets out of the file

A config file belongs in version control; a password does not. Reference an environment variable and resolve it at load time.

ENV_PATTERN = re.compile(r"\$\{([A-Z_][A-Z0-9_]*)\}")


def expand_env(value):
    if isinstance(value, str):
        def sub(m):
            name = m.group(1)
            if name not in os.environ:
                raise KeyError(f"config needs environment variable {name}")
            return os.environ[name]
        return ENV_PATTERN.sub(sub, value)
    if isinstance(value, dict):
        return {k: expand_env(v) for k, v in value.items()}
    if isinstance(value, list):
        return [expand_env(v) for v in value]
    return value


database = yaml.safe_load("""
database:
  host: localhost
  port: 55433
  name: gis
  user: user
  password: ${PARCELS_DB_PASSWORD}    # resolved from the environment
""")
try:
    expand_env(database)
except KeyError as exc:
    print("without the variable:", exc)
os.environ["PARCELS_DB_PASSWORD"] = "guide"
print("with it:", {k: ("***" if k == "password" else v) for k, v in expand_env(database)["database"].items()})

without the variable: 'config needs environment variable PARCELS_DB_PASSWORD'
with it: {'host': 'localhost', 'port': 55433, 'name': 'gis', 'user': 'user', 'password': '***'}

Example 4: Validating with a schema

When configs grow past a dozen keys, hand-written checks get tedious. A dataclass costs nothing extra and turns cfg["params"]["min_area_m2"] into cfg.params.min_area_m2:

from dataclasses import dataclass, field


@dataclass
class Params:
    target_crs: str
    min_area_m2: float = 0.0
    simplify_m: float = 0.0


@dataclass
class Config:
    source: Path
    output_dir: Path
    params: Params
    steps: list = field(default_factory=list)

    @classmethod
    def from_dict(cls, d, base: Path):
        return cls(source=(base / d["input"]["source"]).resolve(),
                   output_dir=(base / d["output"]["dir"]).resolve(),
                   params=Params(**d["params"]), steps=d["steps"])


raw = yaml.safe_load(Path("config/parcels.yml").read_text())
print(Config.from_dict(raw, Path("config")).params)
try:
    Config.from_dict(deep_merge(raw, {"params": {"min_area": 20}}), Path("config"))
except TypeError as exc:
    print("TypeError:", exc)

Params(target_crs='EPSG:27700', min_area_m2=50, simplify_m=0.0)
TypeError: Params.__init__() got an unexpected keyword argument 'min_area'. Did you mean 'min_area_m2'?

An immediate TypeError when the config carries a key the code does not know — here a min_area typo — is the dataclass version of the unknown-key check. What a dataclass does not do is check types: min_area_m2: "fifty" would pass straight through, which is where a validating library such as pydantic earns its place.

Example 5: The entry point

Twenty lines that tie config, validation, pipeline construction, and the run together:

%%writefile run_pipeline.py
import logging
import sys

import geopandas as gpd

from config import load_config
from steps import REGISTRY, apply_all


def main(*config_paths):
    cfg = load_config(*config_paths)
    logging.basicConfig(level=cfg["logging"]["level"], format="%(levelname)-7s %(message)s", stream=sys.stdout)
    log = logging.getLogger(cfg.get("name", "pipeline"))
    log.info("config: %s", cfg["_config_path"])
    steps = [REGISTRY[n] for n in cfg["steps"]]
    gdf = gpd.read_file(cfg["input"]["source"])
    result = apply_all(gdf, steps, cfg["params"], log=log.info)
    cfg["output"]["dir"].mkdir(parents=True, exist_ok=True)
    out = cfg["output"]["dir"] / f"{cfg.get('name', 'parcels')}.gpkg"
    if out.exists() and not cfg["output"]["overwrite"]:
        raise FileExistsError(f"{out.name} exists and overwrite is false")
    result.to_file(out, driver=cfg["output"]["driver"])
    log.info("wrote %d features -> %s", len(result), out.name)


if __name__ == "__main__":
    main(*(sys.argv[1:] or ["config/parcels.yml"]))

!{sys.executable} run_pipeline.py config/parcels.yml 2>&1 | grep -v Created
!{sys.executable} run_pipeline.py config/parcels.yml 2>&1 | tail -1
!{sys.executable} run_pipeline.py config/base.yml config/prod.yml 2>&1 | grep -E "filter_area|wrote"

INFO    config: parcels.yml
INFO    drop_empty    21253 ->  21253
INFO    make_valid    21253 ->  21253
INFO    reproject     21253 ->  21253
INFO    clip          21253 ->   5001
INFO    area           5001 ->   5001
INFO    filter_area    5001 ->   4604
INFO    wrote 4604 features -> parcels.gpkg
FileExistsError: parcels.gpkg exists and overwrite is false
INFO    filter_area    5001 ->   4957
INFO    wrote 4957 features -> parcels.gpkg

The first run writes the output. The second refuses, because the file exists and the config says overwrite: false — a guard that would otherwise have to be remembered. The layered production config writes to its own folder and keeps the slivers between 5 and 50 m² that the default run drops.

A run you can name and repeat

Externalising configuration changes what a "run" is. Before, a run is a state of the source code that existed for a few minutes and is now gone. After, a run is a config file — a small text artefact you can commit, diff, review, attach to a ticket, and hand to someone else. When February's numbers look odd, you diff February's config against January's and the cause is usually right there.

The validation step is where most of the value hides. Pipelines fail expensively: a long reprojection runs, then the write fails because the output driver name was misspelled. Validating up front turns that into a failure in the first second, with a message that says exactly which key is wrong. The rule of thumb — check everything you can before touching data — costs a dozen lines and saves hours.

There is a boundary worth defending. Config should describe which data, which settings, which steps — not what the steps do. The moment a config file starts carrying conditional logic or expressions to be evaluated, you have invented a programming language with no debugger and no tests. Keep behaviour in Python and selection in YAML: a steps: list naming registered functions is on the right side of that line; an if expression embedded in a string is not.

Layering deserves a clear order, because ambiguity here causes real confusion. Defaults in code sit at the bottom, the base config overrides them, an environment config overrides that, and command-line flags win over everything — most specific wins. Write that order in a comment at the top of the loader and never deviate.

Credentials, drift and one config per environment

When the file itself will not load

YAML's own traps — tabs, no parsing as False, an unquoted 27700 arriving as an integer, a duplicate key silently winning — are the subject of YAML config file will not load. Of those, only the type surprises get past the parser, and the collected validation above is what catches them before a feature is read: the target_crs: 27700 in the broken config was one of its three reported problems.

The config file is not a place for credentials

Never commit a password, token, or connection string. Reference an environment variable in the config and resolve it at load time, or read secrets from a separate file that is listed in .gitignore. A pipeline that reads from PostGIS needs this on day one.

Configs drift from the code

Add a required key to the code and every old config becomes invalid — usually discovered at the worst moment. Give new keys a default in the loader so old files keep working, and reserve hard failures for keys that genuinely have no safe default.

One config per environment, not per run

If you find yourself creating parcels-2026-08-01.yml, parcels-2026-08-02.yml, and so on, the date is a parameter, not a configuration. Pass it as a command-line flag and keep one config per environment.

Internal links

• How to Build a GIS Data Pipeline in Python
• Chain GIS Processing Steps into a Reusable Pipeline
• Turn a GIS Script into a Command-Line Tool
• Schedule a Python GIS Script to Run Automatically
• Validate Pipeline Inputs and Outputs Automatically
• YAML Config File Will Not Load: Common Errors in a Python GIS Pipeline
• Make a GIS Workflow Reproducible
• How to Connect GeoPandas to PostGIS

FAQ

Why YAML rather than JSON or TOML for a GIS pipeline config?

YAML supports comments, which matter enormously in a file that documents thresholds and CRS choices, and its nesting is easy to read. JSON has no comments and is unforgiving about trailing commas. TOML is a strong choice for flat configs but gets clumsy with deeply nested sections. Any of the three beats constants in a script.

Is yaml.safe_load() really necessary?

Yes. yaml.load() without a safe loader can instantiate arbitrary Python objects described in the file, which turns any config you did not personally write into code execution. safe_load restricts parsing to plain data types — everything a configuration legitimately needs.

How do I keep database passwords out of the config file?

Put a placeholder like ${PARCELS_DB_PASSWORD} in the YAML and expand it from the environment when loading. The config stays committable and readable, the secret lives in the environment or a secret manager, and a missing variable fails loudly at load time rather than as a confusing connection error later.

Should the list of processing steps live in the config?

Yes, if your steps are in a registry — a list of names is selection, not logic, and it makes reordering a review-able one-line diff. What should not go in the config is the behaviour of a step. If a config entry starts to look like an expression to evaluate, move it back into Python.

How do I handle different settings for development and production?

Keep a base.yml with everything shared, then thin dev.yml and prod.yml files carrying only the differences, and deep-merge them at load time. This keeps the two environments genuinely identical except where you intended a difference — the opposite of maintaining two full copies that quietly diverge.

What happens when a config key is missing?

That depends on whether the key has a sensible default. Provide defaults in code for optional settings so older config files keep working, and raise a clear error for genuinely required keys. Either way, decide at load time: a KeyError deep inside a transform twenty minutes into a run is the worst possible outcome.

How do I record which config a given output came from?

Write the merged config into the run's output folder as config_used.json, next to the data and the log. Because base and overlay are already combined, it records what actually ran rather than what any one file said. The snapshot above keeps paths as written (../data/raw/parcels.gpkg); snapshot the result of load_config() instead if you want them absolute. Do it before expanding environment variables, or the database password from Example 3 ends up on disk. Combine it with the run manifest for a complete record.