⚙️ How-To

How to Chain GIS Processing Steps into a Reusable Pipeline in Python

Most GIS scripts grow the same way. You write a function to reproject, then one to clip, then one to summarise, and before long the bottom of the file is a wall of calls where the output of each is fed into the next by hand. It works — until you want to run the same three steps in a different order, or skip one, or add a fourth for a different project. The fix is small and old: give every step the same shape, then treat the workflow as a list of steps rather than a sequence of calls. This article shows how to do that in plain Python, with no framework.

Problem statement

You have useful GIS functions but no clean way to combine them. The symptoms:

  • The call site is a wall of temporary variables — gdf1, gdf2, gdf_clipped, gdf_final, and one typo away from using the wrong one.
  • Steps cannot be reordered or reused — each function takes slightly different arguments, so swapping two of them means editing both.
  • Copy-paste between projects — the next project needs the same clip-and-summarise pair, so you copy 80 lines and change the paths.
  • Options are tangled into the logic — the buffer distance is a literal in the middle of a function instead of a parameter.
  • No way to see the workflow — nobody can tell what the script does without reading all of it.

The goal: a set of interchangeable steps with one shared contract, composed into a named pipeline you can read at a glance, reorder freely, and reuse in the next project without edits.

Quick answer

Give every step the same signature — take a GeoDataFrame and a config, return a GeoDataFrame — then build the workflow as a list and fold the data through it. Parameterise a step by wrapping it in a factory function that returns the configured step.

Three interchangeable steps, each taking a GeoDataFrame in and returning a GeoDataFrame out.
One contract — GeoDataFrame in, GeoDataFrame out — is what makes steps interchangeable.
from functools import reduce
import geopandas as gpd

# Every step: (gdf, cfg) -> gdf
def reproject(gdf, cfg):
    return gdf.to_crs(cfg["target_crs"])

def drop_empty(gdf, cfg):
    return gdf[gdf.geometry.notna() & ~gdf.geometry.is_empty]

def add_area(gdf, cfg):
    out = gdf.copy()
    out["area_m2"] = out.geometry.area
    return out

def buffer(distance):                     # factory: returns a configured step
    def step(gdf, cfg):
        out = gdf.copy()
        out["geometry"] = out.geometry.buffer(distance)
        return out
    step.__name__ = f"buffer({distance})"
    return step

PIPELINE = [drop_empty, reproject, buffer(25), add_area]

def apply_all(gdf, steps, cfg):
    return reduce(lambda data, step: step(data, cfg), steps, gdf)

cfg = {"target_crs": "EPSG:27700"}
result = apply_all(gpd.read_file("data/raw/sites.gpkg"), PIPELINE, cfg)

PIPELINE is now a readable description of the workflow, and every entry in it is a function you can test on its own.

Step-by-step solution

Agree on one contract

The contract is the whole idea: a step takes a GeoDataFrame and a config and returns a new GeoDataFrame. Nothing else. No printing, no writing files, no reading paths. Steps that obey it can be reordered, dropped, tested, and reused without thought; steps that do not will always need special handling at the call site.

def step(gdf: gpd.GeoDataFrame, cfg: dict) -> gpd.GeoDataFrame:
    ...

Two rules make the contract stick. First, never mutate the input — take a .copy() before assigning a column, or you will corrupt the caller's frame. Second, always return a frame, even when the step is a no-op; a step that returns None on some branch will blow up three steps later, far from the cause.

Write the steps small

A step should do one thing you can name. clip_to_boundary is a step; prepare_data is a bundle of decisions hiding from you. Small steps make the pipeline list read like documentation, and they keep failures precise — when the log says filter_by_area failed, you know exactly where to look.

def clip_to_boundary(gdf, cfg):
    boundary = cfg["boundary"].to_crs(gdf.crs)   # align first, always
    return gpd.clip(gdf, boundary)

def filter_by_area(gdf, cfg):
    return gdf[gdf.geometry.area >= cfg["min_area_m2"]]

def keep_columns(gdf, cfg):
    cols = [*cfg["keep_columns"], gdf.geometry.name]
    return gdf[cols]

Parameterise with factories, not globals

When a step needs its own setting, the temptation is to reach for a module-level constant. Use a factory instead: a function that takes the parameter and returns a step closed over it. The pipeline list then shows the parameter right where it is used.

def simplify(tolerance_m):
    def step(gdf, cfg):
        out = gdf.copy()
        out["geometry"] = out.geometry.simplify(tolerance_m)
        return out
    step.__name__ = f"simplify({tolerance_m}m)"
    return step

PIPELINE = [reproject, simplify(5), add_area]

Setting __name__ matters more than it looks — it is what your logs and error messages will show.

Compose the list, then fold

With a shared contract, running the pipeline is a fold: start with the input frame and apply each step in turn. functools.reduce expresses it in one line, but an explicit loop is just as good and easier to instrument.

Nested function calls on the left versus a readable list of named steps on the right.
Nested calls hide the workflow; a list of steps *is* the workflow.
def apply_all(gdf, steps, cfg, log=None):
    data = gdf
    for step in steps:
        before = len(data)
        data = step(data, cfg)
        if log:
            log.info("%-22s %6d → %6d features", step.__name__, before, len(data))
    return data

That single loop now gives every step timing, feature counts, and a name in the log — without any step knowing about logging.

Name your pipelines and keep them together

Once composition is cheap, having several named pipelines is natural. Keep them in one place so the set of workflows in the project is visible at a glance.

PIPELINES = {
    "clean":   [drop_empty, make_valid_geoms, reproject],
    "analyse": [drop_empty, reproject, clip_to_boundary, filter_by_area, add_area],
    "publish": [reproject_to_wgs84, simplify(2), keep_columns],
}

result = apply_all(gdf, PIPELINES["analyse"], cfg)

This dictionary is also the natural thing to expose to a YAML config or a command-line flag later — --pipeline analyse becomes a one-line lookup.

Code examples

Example 1: A step library you can reuse across projects

Put the generic steps in their own module. They have no project-specific knowledge, so the next project imports them unchanged.

# steps.py
import geopandas as gpd

def drop_empty(gdf, cfg):
    return gdf[gdf.geometry.notna() & ~gdf.geometry.is_empty].copy()

def make_valid_geoms(gdf, cfg):
    out = gdf.copy()
    bad = ~out.geometry.is_valid
    out.loc[bad, "geometry"] = out.loc[bad, "geometry"].make_valid()
    return out

def reproject(gdf, cfg):
    return gdf.to_crs(cfg["target_crs"])

def drop_duplicate_geoms(gdf, cfg):
    return gdf.loc[~gdf.geometry.normalize().duplicated()].copy()

Example 2: Steps that need a second layer

A clip or a join needs another dataset. Put the extra layer in the config rather than changing the step signature — the contract stays intact.

cfg = {
    "target_crs": "EPSG:27700",
    "boundary": gpd.read_file("data/raw/district.geojson"),
    "min_area_m2": 100,
}

def clip_to_boundary(gdf, cfg):
    boundary = cfg["boundary"].to_crs(gdf.crs)
    return gpd.clip(gdf, boundary)

def join_wards(gdf, cfg):
    wards = cfg["wards"].to_crs(gdf.crs)
    return gpd.sjoin(gdf, wards[["ward_name", "geometry"]],
                     how="left", predicate="within").drop(columns="index_right")

Example 3: A registry so pipelines can be named in text

Map step names to functions and a pipeline becomes a list of strings — which is exactly what you need to move it into a config file.

REGISTRY = {
    "drop_empty": drop_empty,
    "make_valid": make_valid_geoms,
    "reproject": reproject,
    "clip": clip_to_boundary,
    "area": add_area,
}

def build(names):
    missing = [n for n in names if n not in REGISTRY]
    if missing:
        raise KeyError(f"unknown steps: {missing}. Known: {sorted(REGISTRY)}")
    return [REGISTRY[n] for n in names]

pipeline = build(["drop_empty", "make_valid", "reproject", "clip", "area"])

Example 4: Guarding the contract in development

A tiny decorator catches the two mistakes every step author eventually makes: returning None, and returning something that is not a GeoDataFrame.

import functools
import geopandas as gpd

def step(fn):
    @functools.wraps(fn)
    def wrapper(gdf, cfg):
        out = fn(gdf, cfg)
        if out is None:
            raise TypeError(f"step '{fn.__name__}' returned None")
        if not isinstance(out, gpd.GeoDataFrame):
            raise TypeError(f"step '{fn.__name__}' returned {type(out).__name__}")
        return out
    return wrapper

@step
def add_area(gdf, cfg):
    out = gdf.copy()
    out["area_m2"] = out.geometry.area
    return out

Example 5: The whole thing, composed and run

Steps, registry, fold, logging — about sixty lines, and it is the backbone of every workflow in the project.

import logging
from functools import reduce
from pathlib import Path
import geopandas as gpd

logging.basicConfig(level=logging.INFO, format="%(message)s")
log = logging.getLogger("chain")

def drop_empty(gdf, cfg):
    return gdf[gdf.geometry.notna() & ~gdf.geometry.is_empty].copy()

def reproject(gdf, cfg):
    return gdf.to_crs(cfg["target_crs"])

def add_area(gdf, cfg):
    out = gdf.copy()
    out["area_m2"] = out.geometry.area
    return out

def filter_min_area(min_m2):
    def s(gdf, cfg):
        return gdf[gdf["area_m2"] >= min_m2].copy()
    s.__name__ = f"filter_min_area({min_m2})"
    return s

PIPELINE = [drop_empty, reproject, add_area, filter_min_area(250)]

def apply_all(gdf, steps, cfg):
    def one(data, step):
        before = len(data)
        out = step(data, cfg)
        log.info("%-24s %6d → %6d", step.__name__, before, len(out))
        return out
    return reduce(one, steps, gdf)

cfg = {"target_crs": "EPSG:27700"}
gdf = gpd.read_file("data/raw/sites.gpkg")
result = apply_all(gdf, PIPELINE, cfg)
Path("data/processed").mkdir(parents=True, exist_ok=True)
result.to_file("data/processed/sites_analysed.gpkg", driver="GPKG")

Explanation

The reason a shared contract pays off so quickly is that it turns n incompatible functions into n interchangeable parts. Without it, combining two steps requires knowing both of their signatures; with it, any step composes with any other, and the workflow becomes data — a list you can build, slice, reverse, or read from a file.

Keeping steps pure — no I/O, no printing, no globals — is what makes them testable. A test for filter_min_area builds a four-row GeoDataFrame in memory, applies the step, and asserts two rows survive. No fixtures, no sample files, no filesystem. That is a test you will actually write, and it will still pass in two years.

A registry mapping step names to functions, letting a text list of names build a runnable pipeline.
A registry is the bridge from code to configuration: names in text, functions in Python.

Factories deserve a word of defence. Closing over a parameter looks like extra ceremony compared with reading it from cfg, and for settings shared by the whole run — the target CRS, the output folder — cfg is the right home. Use a factory when the same step appears twice with different settings: two buffers at 100 m and 500 m in one pipeline are impossible to express with a single config value, and trivial with buffer(100) and buffer(500).

This composition style is deliberately not a framework. There is no scheduler, no dependency graph, no DSL to learn — just functions and a list, which means the whole mechanism fits in one screen and any Python developer can debug it. When the workflow genuinely becomes a graph rather than a line, that is the moment to look at a real orchestrator; until then, the list is both simpler and easier to reason about. The complete pipeline workflow shows how the same idea scales up to extract-and-load stages either side of these transforms.

Edge cases or notes

A step that must not run in the middle

Reading the source and writing the output are not transform steps — they do not obey the contract, and forcing them into it produces steps that lie about their return values. Keep extract and load as bookends around the chain, as in the complete pipeline workflow.

Steps that change the CRS

Any step that reprojects changes an invariant every later step depends on. Either put reprojection first and treat the CRS as fixed for the rest of the chain, or have measurement steps assert the CRS they need: assert gdf.crs.is_projected, "reproject before measuring". Silent CRS drift is the most common cause of results that look plausible and are wrong.

Steps that drop every feature

A filter with a wrong threshold returns an empty frame, and every later step happily processes zero rows. Log feature counts at every step — as the runner above does — and the collapse is obvious in the log rather than invisible in the output.

Mutating the input by accident

gdf["col"] = … modifies the frame in place, so a step without a .copy() changes its caller's data. That breaks the mental model of a chain and produces bugs that move when you reorder steps. Copy at the top of any step that assigns.

Performance of copying at every step

Copying a large GeoDataFrame in every step costs memory and time. For most workflows this is irrelevant; for large data, copy once at the start of the chain and let the steps mutate deliberately, or drop columns early so what you copy is small. Measure before optimising — the reprojection is almost always the expensive part, not the copies. See speed up GeoPandas for large datasets.

Order-dependent steps

Some pairs genuinely cannot be reordered: you must repair geometries before dissolving, and reproject before measuring area. Encode those dependencies as assertions inside the steps rather than as comments, so a wrong order fails immediately with a clear message instead of producing quiet nonsense.

FAQ

What should the signature of a pipeline step be?

(gdf, cfg) -> gdf. The GeoDataFrame is the data flowing through; the config carries settings and any extra layers the step needs. Keeping every step to that one signature is what makes them interchangeable — and it costs nothing, because a step that ignores cfg simply does not use it.

How do I pass a parameter to a single step?

Use a factory: a function that takes the parameter and returns the step, closed over it. buffer(25) returns a step that buffers by 25 metres. This keeps the parameter visible in the pipeline list and lets the same step appear twice with different settings — impossible with a single config key.

Should steps write files?

No. A step that writes to disk cannot be reordered, cannot be tested without a filesystem, and hides an effect the pipeline list does not show. Keep reading and writing in dedicated extract and load stages at either end of the chain, and let every step in the middle be a pure transform.

How do I skip a step conditionally?

Build the list conditionally rather than putting an if inside the step: steps = [drop_empty, reproject] + ([clip] if cfg.get("boundary") else []) + [add_area]. The pipeline still reads as a list, and the condition is visible where the workflow is described instead of buried in a function.

Can I run steps in parallel?

Steps in a chain are sequential by definition — each needs the previous one's output. Parallelism belongs inside a step, across features or files: split the frame, map the work over a pool, concatenate the results. See speed up batch GIS jobs with parallel processing for the pattern and its caveats.

How is this different from just calling the functions in order?

Mechanically, very little — and that is the point. What changes is that the workflow becomes a value you can name, store, reorder, log, and read from a config file, instead of control flow buried in a script. The fold gives you one place to add logging, timing, or validation for every step at once.

Where does error handling go?

In the runner, not the steps. Wrap the loop so a failure is reported with the step's name attached, and let steps raise plain exceptions describing what went wrong. That keeps each step readable and gives every failure a consistent, useful message. For flaky steps that deserve another attempt, see add retries and timeouts.