⚙️ How-To

How to Drive a GIS Pipeline from a YAML Config File in Python

The clearest sign that a script has outgrown itself is a block of constants at the top that you edit before every run. Change the input folder, change the CRS, comment out the clip, change it back. Each edit is a chance to break something, and none of it is recorded anywhere. Moving those settings into a YAML file changes the economics completely: the code stops changing, the config becomes the thing you version, and running last month's job again is a matter of pointing at last month's file.

Problem statement

Your pipeline works but every run needs a code edit. Concretely:

  • Editing code to change a path — the difference between the January and February run is two string literals buried in a script.
  • No record of what a run used — the settings that produced an output are gone the moment you edit them for the next run.
  • Colleagues cannot run it — their data lives elsewhere, so they must understand your Python before they can change a folder.
  • Dev and production drift — you keep two copies of the script with slightly different constants, and they diverge.
  • Secrets in the repository — a database password sitting in a committed .py file.

The goal: one config file per scenario, validated before the run starts, with the code identical in every environment and the exact settings of each run saved alongside its output.

Quick answer

Put paths, parameters, and the step list in a YAML file. Load it with yaml.safe_load(), validate the keys and turn path strings into Path objects immediately, and pass the resulting dictionary into the pipeline you already have.

A YAML file loaded into a dictionary, validated, resolved to paths, and handed to the pipeline runner.
Load, validate, resolve, run — validation belongs before the first byte of data is read.
# config/parcels.yml
input:
  source: data/raw/parcels.gpkg
  boundary: data/raw/district.geojson
output:
  dir: data/processed
  driver: GPKG
params:
  target_crs: EPSG:27700
  min_area_m2: 50
steps: [drop_empty, make_valid, reproject, clip, area, filter_area]
import sys
from pathlib import Path
import yaml

REQUIRED = {"input", "output", "params", "steps"}

def load_config(path):
    cfg = yaml.safe_load(Path(path).read_text())
    missing = REQUIRED - cfg.keys()
    if missing:
        raise ValueError(f"config is missing sections: {sorted(missing)}")
    cfg["input"] = {k: Path(v) for k, v in cfg["input"].items()}
    cfg["output"]["dir"] = Path(cfg["output"]["dir"])
    for name, p in cfg["input"].items():
        if not p.exists():
            raise FileNotFoundError(f"input '{name}' not found: {p}")
    return cfg

if __name__ == "__main__":
    cfg = load_config(sys.argv[1])
    run(cfg)                       # your existing pipeline, unchanged

python run_pipeline.py config/parcels.yml now runs the January job; a second file runs February's. The code never changes.

Step-by-step solution

Choose YAML, and load it safely

YAML wins for config that humans edit: comments, no trailing-comma traps, readable nesting, and native types for numbers and booleans. JSON is fine but cannot carry comments, and TOML is excellent but awkward for deeply nested structures.

import yaml
from pathlib import Path

cfg = yaml.safe_load(Path("config/parcels.yml").read_text())

Always safe_load, never load. Plain yaml.load() can construct arbitrary Python objects from a file, which is a real risk the moment a config comes from anywhere but your own hands. safe_load gives you dictionaries, lists, strings, numbers, and booleans — everything a config should contain.

Give the file a shape and keep to it

An unstructured config becomes a junk drawer. Four top-level sections cover almost every pipeline, and the grouping tells a reader what each key is for.

input:    # where data comes from
output:   # where results go
params:   # the numbers and codes that tune the run
steps:    # which transforms to apply, in order

Validate before you read a single feature

A config error should stop the run in the first second, not after a twenty-minute reprojection. Check required sections, check that inputs exist, and check that values are the type you expect.

def validate(cfg):
    problems = []
    for section in ("input", "output", "params", "steps"):
        if section not in cfg:
            problems.append(f"missing section: {section}")
    if problems:
        raise ValueError("; ".join(problems))

    for name, path in cfg["input"].items():
        if not Path(path).exists():
            problems.append(f"input '{name}' does not exist: {path}")

    crs = cfg["params"].get("target_crs", "")
    if not str(crs).upper().startswith("EPSG:"):
        problems.append(f"target_crs should look like 'EPSG:27700', got {crs!r}")

    if not isinstance(cfg["steps"], list) or not cfg["steps"]:
        problems.append("steps must be a non-empty list")

    if problems:
        raise ValueError("Invalid config:\n  - " + "\n  - ".join(problems))
    return cfg

Collecting all the problems and reporting them together is worth the extra few lines. Fixing five mistakes in one pass beats five run-fail-edit cycles.

Resolve paths relative to the config file

A config that only works from one working directory is a config that will fail under a scheduler. Resolve every relative path against the location of the config file itself, so the same file works from anywhere.

def load_config(path):
    path = Path(path).resolve()
    cfg = yaml.safe_load(path.read_text())
    base = path.parent
    cfg["input"] = {k: (base / v).resolve() for k, v in cfg["input"].items()}
    cfg["output"]["dir"] = (base / cfg["output"]["dir"]).resolve()
    return cfg

This single detail removes the most common cause of "it works when I run it, but the cron job fails" — see scheduling a Python GIS script.

Turn the step list into functions

If your steps live in a registry, the steps: list in YAML becomes a runnable pipeline with three lines — and an unknown step name fails immediately with a helpful message.

from steps import REGISTRY

def build_pipeline(names):
    unknown = [n for n in names if n not in REGISTRY]
    if unknown:
        raise KeyError(f"unknown steps {unknown}; known steps: {sorted(REGISTRY)}")
    return [REGISTRY[n] for n in names]

Layer configs instead of duplicating them

Two environments should not mean two copies of every setting. Keep a base.yml with everything shared and small overlay files with only the differences, then merge.

A base config overlaid by an environment config and then by command-line flags, each layer overriding the one below.
Defaults at the bottom, the most specific override at the top — one predictable order.
import copy

def deep_merge(base, override):
    out = copy.deepcopy(base)
    for key, value in override.items():
        if isinstance(value, dict) and isinstance(out.get(key), dict):
            out[key] = deep_merge(out[key], value)
        else:
            out[key] = value
    return out

base = yaml.safe_load(Path("config/base.yml").read_text())
prod = yaml.safe_load(Path("config/prod.yml").read_text())
cfg = deep_merge(base, prod)

Save the config with the output

The final habit that makes configs pay off: copy the resolved config into the run's output folder. Months later, the folder answers its own questions about what produced it.

import json

def snapshot(cfg, run_dir):
    serialisable = json.loads(json.dumps(cfg, default=str))   # Paths → strings
    (run_dir / "config_used.json").write_text(json.dumps(serialisable, indent=2))

Code examples

Example 1: A complete config file

Every setting the pipeline needs, grouped, commented, and readable by someone who does not know Python.

# config/parcels-prod.yml
name: parcels-monthly

input:
  source: ../data/raw/parcels.gpkg
  boundary: ../data/raw/district.geojson

output:
  dir: ../data/processed
  driver: GPKG
  overwrite: false

params:
  target_crs: EPSG:27700     # British National Grid — metres
  min_area_m2: 50            # drop slivers below this
  simplify_m: 0              # 0 disables simplification

steps:
  - drop_empty
  - make_valid
  - reproject
  - clip
  - area
  - filter_area

logging:
  level: INFO
  file: ../logs/parcels.log

Example 2: Loader with defaults, merging, and validation

Defaults in code mean a short config file stays valid when you add a new setting.

from pathlib import Path
import copy, yaml

DEFAULTS = {
    "output": {"driver": "GPKG", "overwrite": False},
    "params": {"min_area_m2": 0, "simplify_m": 0},
    "logging": {"level": "INFO", "file": None},
}

def deep_merge(base, override):
    out = copy.deepcopy(base)
    for k, v in (override or {}).items():
        out[k] = deep_merge(out[k], v) if isinstance(v, dict) and isinstance(out.get(k), dict) else v
    return out

def load_config(path):
    path = Path(path).resolve()
    raw = yaml.safe_load(path.read_text()) or {}
    cfg = deep_merge(DEFAULTS, raw)
    base = path.parent
    cfg["input"] = {k: (base / v).resolve() for k, v in cfg["input"].items()}
    cfg["output"]["dir"] = (base / cfg["output"]["dir"]).resolve()
    cfg["_config_path"] = str(path)
    return validate(cfg)

Example 3: Keeping secrets out of the file

A config file belongs in version control; a password does not. Reference an environment variable and resolve it at load time.

database:
  host: gis-db.internal
  name: parcels
  user: pipeline
  password: ${PARCELS_DB_PASSWORD}    # resolved from the environment
import os, re

ENV_PATTERN = re.compile(r"\$\{([A-Z_][A-Z0-9_]*)\}")

def expand_env(value):
    if isinstance(value, str):
        def sub(m):
            name = m.group(1)
            if name not in os.environ:
                raise KeyError(f"config needs environment variable {name}")
            return os.environ[name]
        return ENV_PATTERN.sub(sub, value)
    if isinstance(value, dict):
        return {k: expand_env(v) for k, v in value.items()}
    if isinstance(value, list):
        return [expand_env(v) for v in value]
    return value

Example 4: Validating with a schema

When configs grow past a dozen keys, hand-written checks get tedious. A schema library gives you types, defaults, and clear error messages for free.

from dataclasses import dataclass, field
from pathlib import Path

@dataclass
class Params:
    target_crs: str
    min_area_m2: float = 0.0
    simplify_m: float = 0.0

@dataclass
class Config:
    source: Path
    output_dir: Path
    params: Params
    steps: list = field(default_factory=list)

    @classmethod
    def from_dict(cls, d, base: Path):
        return cls(
            source=(base / d["input"]["source"]).resolve(),
            output_dir=(base / d["output"]["dir"]).resolve(),
            params=Params(**d["params"]),
            steps=d["steps"],
        )

A dataclass costs nothing extra and turns cfg["params"]["min_area_m2"] into cfg.params.min_area_m2 — with an immediate TypeError if the config carries a key the code does not know.

Example 5: The entry point

Twenty lines that tie config, validation, pipeline construction, and the run together.

import logging, sys
from pathlib import Path
from config import load_config
from steps import REGISTRY
from pipeline import apply_all
import geopandas as gpd

def main(config_path):
    cfg = load_config(config_path)
    logging.basicConfig(level=cfg["logging"]["level"],
                        format="%(asctime)s %(levelname)-7s %(message)s")
    log = logging.getLogger(cfg.get("name", "pipeline"))
    log.info("config: %s", cfg["_config_path"])

    steps = [REGISTRY[n] for n in cfg["steps"]]
    gdf = gpd.read_file(cfg["input"]["source"])
    result = apply_all(gdf, steps, cfg["params"])

    cfg["output"]["dir"].mkdir(parents=True, exist_ok=True)
    out = cfg["output"]["dir"] / f"{cfg.get('name', 'output')}.gpkg"
    if out.exists() and not cfg["output"]["overwrite"]:
        raise FileExistsError(f"{out} exists and overwrite is false")
    result.to_file(out, driver=cfg["output"]["driver"])
    log.info("wrote %d features → %s", len(result), out)

if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "config/parcels.yml")

Explanation

Externalising configuration changes what a "run" is. Before, a run is a state of the source code that existed for a few minutes and is now gone. After, a run is a config file — a small text artefact you can commit, diff, review, attach to a ticket, and hand to someone else. When February's numbers look odd, you diff February's config against January's and the cause is usually right there.

The validation step is where most of the value hides. Pipelines fail expensively: the reprojection runs for fifteen minutes, then the write fails because the output driver name was misspelled. Validating up front turns a twenty-minute failure into a one-second one with a message that says exactly which key is wrong. The rule of thumb — check everything you can before touching data — costs a dozen lines and saves hours.

Hard-coded constants edited before every run versus one script driven by several config files.
One script, many configs: the code stops changing and the runs become reviewable artefacts.

There is a boundary worth defending. Config should describe which data, which settings, which steps — not what the steps do. The moment a config file starts carrying conditional logic or expressions to be evaluated, you have invented a programming language with no debugger and no tests. Keep behaviour in Python and selection in YAML: a steps: list naming registered functions is on the right side of that line; an if expression embedded in a string is not.

Layering deserves a clear order, because ambiguity here causes real confusion. Defaults in code sit at the bottom, the base config overrides them, an environment config overrides that, and command-line flags win over everything — most specific wins. Write that order in a comment at the top of the loader and never deviate.

Edge cases or notes

YAML's surprising types

no parses as boolean False, on as True, and an unquoted EPSG:27700 is a plain string but 27700 is an integer. Version numbers like 3.10 become floats and lose the trailing zero. Quote anything that must stay a string, and validate types rather than trusting the parser.

Tabs are not allowed

YAML forbids tab characters for indentation, and the resulting parse error can be baffling because tabs are invisible. If a file that looks correct will not parse, check for tabs first — most editors can show whitespace or convert on save.

Relative paths and the working directory

A relative path in a config is resolved against something, and it must not be the current working directory, which changes depending on where the job is launched from. Resolve against the config file's own folder, as shown above, and the same file works from your editor, a terminal, and cron alike.

The config file is not a place for credentials

Never commit a password, token, or connection string. Reference an environment variable in the config and resolve it at load time, or read secrets from a separate file that is listed in .gitignore. A pipeline that reads from PostGIS needs this on day one.

Configs drift from the code

Add a required key to the code and every old config becomes invalid — usually discovered at the worst moment. Give new keys a default in the loader so old files keep working, and reserve hard failures for keys that genuinely have no safe default.

One config per environment, not per run

If you find yourself creating parcels-2026-08-01.yml, parcels-2026-08-02.yml, and so on, the date is a parameter, not a configuration. Pass it as a command-line flag and keep one config per environment.

FAQ

Why YAML rather than JSON or TOML for a GIS pipeline config?

YAML supports comments, which matter enormously in a file that documents thresholds and CRS choices, and its nesting is easy to read. JSON has no comments and is unforgiving about trailing commas. TOML is a strong choice for flat configs but gets clumsy with deeply nested sections. Any of the three beats constants in a script.

Is yaml.safe_load() really necessary?

Yes. yaml.load() without a safe loader can instantiate arbitrary Python objects described in the file, which turns any config you did not personally write into code execution. safe_load restricts parsing to plain data types — everything a configuration legitimately needs.

How do I keep database passwords out of the config file?

Put a placeholder like ${PARCELS_DB_PASSWORD} in the YAML and expand it from the environment when loading. The config stays committable and readable, the secret lives in the environment or a secret manager, and a missing variable fails loudly at load time rather than as a confusing connection error later.

Should the list of processing steps live in the config?

Yes, if your steps are in a registry — a list of names is selection, not logic, and it makes reordering a review-able one-line diff. What should not go in the config is the behaviour of a step. If a config entry starts to look like an expression to evaluate, move it back into Python.

How do I handle different settings for development and production?

Keep a base.yml with everything shared, then thin dev.yml and prod.yml files carrying only the differences, and deep-merge them at load time. This keeps the two environments genuinely identical except where you intended a difference — the opposite of maintaining two full copies that quietly diverge.

What happens when a config key is missing?

That depends on whether the key has a sensible default. Provide defaults in code for optional settings so older config files keep working, and raise a clear error for genuinely required keys. Either way, decide at load time: a KeyError deep inside a transform twenty minutes into a run is the worst possible outcome.

How do I record which config a given output came from?

Write the fully resolved config into the run's output folder as config_used.json, next to the data and the log. Because it is the resolved version — merged, with paths absolute and environment variables expanded — it records what actually ran, not what the file said before merging. Combine it with the run manifest for a complete record.