How to Chain GIS Processing Steps into a Reusable Pipeline in Python

On this page

Most GIS scripts grow the same way. You write a function to reproject, then one to clip, then one to summarise, and before long the bottom of the file is a wall of calls where the output of each is fed into the next by hand. It works β€” until you want to run the same three steps in a different order, or skip one, or add a fourth for a different project. The fix is small and old: give every step the same shape, then treat the workflow as a list of steps rather than a sequence of calls. This article shows how to do that in plain Python, with no framework.

Problem statement

Bristol's 275 OpenStreetMap parks go through five steps β€” drop empty geometries, reproject to British National Grid, clip to the city, drop anything under 100 mΒ², add an area column β€” and come out as 190. The clip removes 54, the area floor 31. When each step is a separate call with its own arguments, those two numbers are invisible, and swapping the clip and the area filter means editing both functions and the variables between them.

Give every step the same signature, step(gdf, cfg) -> gdf, and the workflow becomes a list. A loop over that list can log each step's count in and out, which is where the 54 and the 31 come from. The same list can be built from step names in a config file: a misspelt reprojct then fails before any data is read, naming the five steps that exist.

Quick answer

Give every step the same signature β€” take a GeoDataFrame and a config, return a GeoDataFrame β€” then build the workflow as a list and fold the data through it. Parameterise a step by wrapping it in a factory function that returns the configured step.

Three interchangeable steps, each taking a GeoDataFrame and returning one. (gdf, cfg) β†’ gdfdrop_emptygdf β†’ gdfreprojectgdf β†’ gdfbuffer(25)gdf β†’ gdfadd_areagdf β†’ gdfOne shared contract turns n functions into n interchangeable parts spatialworkflow.io
One contract β€” GeoDataFrame in, GeoDataFrame out β€” is what makes steps interchangeable.
from functools import reduce
import geopandas as gpd

# Every step: (gdf, cfg) -> gdf
def reproject(gdf, cfg):
    return gdf.to_crs(cfg["target_crs"])

def drop_empty(gdf, cfg):
    return gdf[gdf.geometry.notna() & ~gdf.geometry.is_empty]

def add_area(gdf, cfg):
    out = gdf.copy()
    out["area_m2"] = out.geometry.area
    return out

def buffer(distance):                     # factory: returns a configured step
    def step(gdf, cfg):
        out = gdf.copy()
        out["geometry"] = out.geometry.buffer(distance)
        return out
    step.__name__ = f"buffer({distance})"
    return step

PIPELINE = [drop_empty, reproject, buffer(25), add_area]

def apply_all(gdf, steps, cfg):
    return reduce(lambda data, step: step(data, cfg), steps, gdf)

cfg = {"target_crs": "EPSG:27700"}
result = apply_all(gpd.read_file("data/raw/sites.gpkg"), PIPELINE, cfg)
print(len(result), "parks,", result.crs.name, "| median buffered area", f"{result.area_m2.median():,.0f} mΒ²")
275 parks, OSGB36 / British National Grid | median buffered area 18,443 mΒ²

PIPELINE is now a readable description of the workflow, and every entry in it is a function you can test on its own.

One contract, small steps, then fold

Agree on one contract

The contract is the whole idea: a step takes a GeoDataFrame and a config and returns a new GeoDataFrame. Nothing else. No printing, no writing files, no reading paths. Steps that obey it can be reordered, dropped, tested, and reused without thought; steps that do not will always need special handling at the call site.

def step(gdf: gpd.GeoDataFrame, cfg: dict) -> gpd.GeoDataFrame:
    ...

Two rules make the contract stick. First, never mutate the input β€” take a .copy() before assigning a column, or you will corrupt the caller's frame. Second, always return a frame, even when the step is a no-op; a step that returns None on some branch will blow up three steps later, far from the cause.

Write the steps small

A step should do one thing you can name. clip_to_boundary is a step; prepare_data is a bundle of decisions hiding from you. Small steps make the pipeline list read like documentation, and they keep failures precise β€” when the log says filter_by_area failed, you know exactly where to look.

def clip_to_boundary(gdf, cfg):
    boundary = cfg["boundary"].to_crs(gdf.crs)   # align first, always
    return gpd.clip(gdf, boundary)

def filter_by_area(gdf, cfg):
    return gdf[gdf.geometry.area >= cfg["min_area_m2"]]

def keep_columns(gdf, cfg):
    cols = [*cfg["keep_columns"], gdf.geometry.name]
    return gdf[cols]

Parameterise with factories, not globals

When a step needs its own setting, the temptation is to reach for a module-level constant. Use a factory instead: a function that takes the parameter and returns a step closed over it. The pipeline list then shows the parameter right where it is used.

def simplify(tolerance_m):
    def step(gdf, cfg):
        out = gdf.copy()
        out["geometry"] = out.geometry.simplify(tolerance_m)
        return out
    step.__name__ = f"simplify({tolerance_m}m)"
    return step

PIPELINE = [reproject, simplify(5), add_area]

Setting __name__ matters more than it looks β€” it is what your logs and error messages will show.

Compose the list, then fold

With a shared contract, running the pipeline is a fold: start with the input frame and apply each step in turn. functools.reduce expresses it in one line, but an explicit loop is just as good and easier to instrument.

Nested function calls versus a readable list of named steps. nesting hides the workflowNested callsβœ•add_area(buffer(reproject(gdf)))βœ•read inside-out to follow itβœ•reordering means re-nestingβœ•no place to hook loggingA list of stepsβœ“PIPELINE = [drop_empty,βœ“ reproject, buffer(25), add_area]βœ“reorder by moving a lineβœ“one loop logs every stepA list of steps IS the workflow β€” and it is data you can build spatialworkflow.io
Nested calls hide the workflow; a list of steps *is* the workflow.
import logging, sys

def apply_all(gdf, steps, cfg, log=None):
    data = gdf
    for step in steps:
        before = len(data)
        data = step(data, cfg)
        if log:
            log.info("%-22s %6d β†’ %6d features", step.__name__, before, len(data))
    return data

logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout)
_ = apply_all(gpd.read_file("data/raw/sites.gpkg"), [drop_empty, reproject, buffer(25), add_area],
              cfg, log=logging.getLogger("chain"))
drop_empty                275 β†’    275 features
reproject                 275 β†’    275 features
buffer(25)                275 β†’    275 features
add_area                  275 β†’    275 features

That single loop now gives every step timing, feature counts, and a name in the log β€” without any step knowing about logging.

Name your pipelines and keep them together

Once composition is cheap, having several named pipelines is natural. Keep them in one place so the set of workflows in the project is visible at a glance.

def make_valid_geoms(gdf, cfg):
    out = gdf.copy()
    bad = ~out.geometry.is_valid
    out.loc[bad, "geometry"] = out.loc[bad, "geometry"].make_valid()
    return out

def reproject_to_wgs84(gdf, cfg):
    return gdf.to_crs(4326)

PIPELINES = {
    "clean":   [drop_empty, make_valid_geoms, reproject],
    "analyse": [drop_empty, reproject, clip_to_boundary, filter_by_area, add_area],
    "publish": [reproject_to_wgs84, simplify(2), keep_columns],
}

logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout, force=True)
gdf = gpd.read_file("data/raw/sites.gpkg")
cfg = {"target_crs": "EPSG:27700", "boundary": gpd.read_file("data/raw/district.geojson"),
       "min_area_m2": 1000, "keep_columns": ["name"]}
result = apply_all(gdf, PIPELINES["analyse"], cfg, log=logging.getLogger("chain"))
drop_empty                275 β†’    275 features
reproject                 275 β†’    275 features
clip_to_boundary          275 β†’    221 features
filter_by_area            221 β†’    190 features
add_area                  190 β†’    190 features

The per-step counts are the point of logging them: 54 parks lie outside the city boundary, in the neighbouring authorities the search box reaches into, and 31 more are under the 1,000 mΒ² floor. Without the log, the output would just be 190 rows.

This dictionary is also the natural thing to expose to a YAML config or a command-line flag later β€” --pipeline analyse becomes a one-line lookup.

A reusable step library on Bristol's parks

The data used on this page

Real layers for Bristol. data/raw/sites.gpkg is the city's parks from OpenStreetMap (Β© OpenStreetMap contributors, ODbL) as they come, in WGS84. data/raw/district.geojson is the City of Bristol from the ONS Local Authority Districts (May 2024) boundaries and data/raw/wards.gpkg its 34 wards from the ONS Wards (December 2024) boundaries, both in the British National Grid.

import json
import zipfile
from pathlib import Path

import geopandas as gpd
import requests
from shapely.geometry import Polygon

ONS = "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/"
raw = Path("data/raw")
raw.mkdir(parents=True, exist_ok=True)
if not (raw / "sites.gpkg").exists():
    cache = Path("osm_parks.json")
    if not cache.exists():
        # live: query = '[out:json][timeout:120];way["leisure"="park"](51.39,-2.73,51.55,-2.50);out geom;'
        #       requests.post("https://overpass-api.de/api/interpreter", data={"data": query}, timeout=180)
        # read instead as Overpass answered that query on 10 October 2026, so the parks stay put
        reply = requests.get("https://data.spatialworkflow.io/osm-bristol-layers-2026-10-10.zip", timeout=120)
        reply.raise_for_status()
        Path("osm-bristol-layers.zip").write_bytes(reply.content)
        zipfile.ZipFile("osm-bristol-layers.zip").extract(cache.name)
    rows = []
    for e in json.loads(cache.read_text())["elements"]:
        xy = [(p["lon"], p["lat"]) for p in e["geometry"]]
        closed = len(xy) > 3 and xy[0] == xy[-1]
        rows.append({"osm_id": e["id"], "name": e.get("tags", {}).get("name"),
                     "geometry": Polygon(xy) if closed else None})       # an open way is not an area
    gpd.GeoDataFrame(rows, crs=4326).to_file(raw / "sites.gpkg")
    gpd.read_file(ONS + "Local_Authority_Districts_May_2024_Boundaries_UK_BFC/FeatureServer/0/query"
                  "?where=LAD24CD%3D%27E06000023%27&outFields=LAD24NM&outSR=27700&f=geojson"
                  ).to_file(raw / "district.geojson")
    gpd.read_file(ONS + "Wards_December_2024_Boundaries_UK_BGC/FeatureServer/0/query"
                  "?where=LAD24CD%3D%27E06000023%27&outFields=WD24NM&outSR=27700&f=geojson"
                  ).rename(columns={"WD24NM": "ward_name"}).to_file(raw / "wards.gpkg")
sites = gpd.read_file(raw / "sites.gpkg")
print(f"{len(sites)} parks, {sites.geometry.isna().sum()} with no geometry, "
      f"{(~sites.geometry.dropna().is_valid).sum()} invalid; "
      f"{len(gpd.read_file(raw / 'wards.gpkg'))} wards")
275 parks, 0 with no geometry, 0 invalid; 34 wards

Example 1: A step library you can reuse across projects

Put the generic steps in their own module. They have no project-specific knowledge, so the next project imports them unchanged.

# steps.py
import geopandas as gpd

def drop_empty(gdf, cfg):
    return gdf[gdf.geometry.notna() & ~gdf.geometry.is_empty].copy()

def make_valid_geoms(gdf, cfg):
    out = gdf.copy()
    bad = ~out.geometry.is_valid
    out.loc[bad, "geometry"] = out.loc[bad, "geometry"].make_valid()
    return out

def reproject(gdf, cfg):
    return gdf.to_crs(cfg["target_crs"])

def drop_duplicate_geoms(gdf, cfg):
    return gdf.loc[~gdf.geometry.normalize().duplicated()].copy()

Example 2: Steps that need a second layer

A clip or a join needs another dataset. Put the extra layer in the config rather than changing the step signature β€” the contract stays intact.

cfg = {
    "target_crs": "EPSG:27700",
    "boundary": gpd.read_file("data/raw/district.geojson"),
    "min_area_m2": 100,
    "wards": gpd.read_file("data/raw/wards.gpkg"),
}

def clip_to_boundary(gdf, cfg):
    boundary = cfg["boundary"].to_crs(gdf.crs)
    return gpd.clip(gdf, boundary)

def join_wards(gdf, cfg):
    wards = cfg["wards"].to_crs(gdf.crs)
    return gpd.sjoin(gdf, wards[["ward_name", "geometry"]],
                     how="left", predicate="within").drop(columns="index_right")

joined = join_wards(result, cfg)
print(f"{joined['ward_name'].notna().sum()} of {len(joined)} parks inside one ward")
163 of 190 parks inside one ward

Example 3: A registry so pipelines can be named in text

Map step names to functions and a pipeline becomes a list of strings β€” which is exactly what you need to move it into a config file.

REGISTRY = {
    "drop_empty": drop_empty,
    "make_valid": make_valid_geoms,
    "reproject": reproject,
    "clip": clip_to_boundary,
    "area": add_area,
}

def build(names):
    missing = [n for n in names if n not in REGISTRY]
    if missing:
        raise KeyError(f"unknown steps: {missing}. Known: {sorted(REGISTRY)}")
    return [REGISTRY[n] for n in names]

pipeline = build(["drop_empty", "make_valid", "reproject", "clip", "area"])
print([f.__name__ for f in pipeline])
try:
    build(["drop_empty", "reprojct"])
except KeyError as exc:
    print(exc)
['drop_empty', 'make_valid_geoms', 'reproject', 'clip_to_boundary', 'add_area']
"unknown steps: ['reprojct']. Known: ['area', 'clip', 'drop_empty', 'make_valid', 'reproject']"

Example 4: Guarding the contract in development

A tiny decorator catches the two mistakes every step author eventually makes: returning None, and returning something that is not a GeoDataFrame.

import functools
import geopandas as gpd

def step(fn):
    @functools.wraps(fn)
    def wrapper(gdf, cfg):
        out = fn(gdf, cfg)
        if out is None:
            raise TypeError(f"step '{fn.__name__}' returned None")
        if not isinstance(out, gpd.GeoDataFrame):
            raise TypeError(f"step '{fn.__name__}' returned {type(out).__name__}")
        return out
    return wrapper

@step
def add_area(gdf, cfg):
    out = gdf.copy()
    out["area_m2"] = out.geometry.area
    return out
@step
def forgot_return(gdf, cfg):
    gdf["area_m2"] = gdf.geometry.area          # mutates, and falls off the end

try:
    apply_all(result, [forgot_return, add_area], cfg)
except TypeError as exc:
    print(exc)
step 'forgot_return' returned None

Example 5: The whole thing, composed and run

Steps, registry, fold, logging β€” about sixty lines, and it is the backbone of every workflow in the project.

import logging
import sys
from functools import reduce
from pathlib import Path
import geopandas as gpd

logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout, force=True)
log = logging.getLogger("chain")

def drop_empty(gdf, cfg):
    return gdf[gdf.geometry.notna() & ~gdf.geometry.is_empty].copy()

def reproject(gdf, cfg):
    return gdf.to_crs(cfg["target_crs"])

def add_area(gdf, cfg):
    out = gdf.copy()
    out["area_m2"] = out.geometry.area
    return out

def filter_min_area(min_m2):
    def s(gdf, cfg):
        return gdf[gdf["area_m2"] >= min_m2].copy()
    s.__name__ = f"filter_min_area({min_m2})"
    return s

PIPELINE = [drop_empty, reproject, add_area, filter_min_area(250)]

def apply_all(gdf, steps, cfg):
    def one(data, step):
        before = len(data)
        out = step(data, cfg)
        log.info("%-24s %6d β†’ %6d", step.__name__, before, len(out))
        return out
    return reduce(one, steps, gdf)

cfg = {"target_crs": "EPSG:27700"}
gdf = gpd.read_file("data/raw/sites.gpkg")
result = apply_all(gdf, PIPELINE, cfg)
Path("data/processed").mkdir(parents=True, exist_ok=True)
result.to_file("data/processed/sites_analysed.gpkg", driver="GPKG")
drop_empty                  275 β†’    275
reproject                   275 β†’    275
add_area                    275 β†’    275
filter_min_area(250)        275 β†’    270
Created 270 records

The last line is not ours: setting the root logger to INFO also turns on the libraries' own messages, here pyogrio reporting the write. Configure a named logger's level instead of the root's if you want only the pipeline's lines.

Why a shared contract turns functions into parts

The reason a shared contract pays off so quickly is that it turns n incompatible functions into n interchangeable parts. Without it, combining two steps requires knowing both of their signatures; with it, any step composes with any other, and the workflow becomes data β€” a list you can build, slice, reverse, or read from a file.

Keeping steps pure β€” no I/O, no printing, no globals β€” is what makes them testable. A test for filter_min_area builds a four-row GeoDataFrame in memory, applies the step, and asserts two rows survive. No fixtures, no sample files, no filesystem. That is a test you will actually write, and it will still pass in two years.

A registry mapping step names in text to Python functions. the bridge from config to codename in YAMLREGISTRY lookupfunction"drop_empty"steps[0]REGISTRY[name]βœ“drop_empty()"reproject"steps[1]REGISTRY[name]βœ“reproject()"clip"steps[2]REGISTRY[name]βœ“clip_to_boundary()An unknown name fails immediately, with the known steps listed spatialworkflow.io
A registry is the bridge from code to configuration: names in text, functions in Python.

Factories deserve a word of defence. Closing over a parameter looks like extra ceremony compared with reading it from cfg, and for settings shared by the whole run β€” the target CRS, the output folder β€” cfg is the right home. Use a factory when the same step appears twice with different settings: two buffers at 100 m and 500 m in one pipeline are impossible to express with a single config value, and trivial with buffer(100) and buffer(500).

This composition style is deliberately not a framework. There is no scheduler, no dependency graph, no DSL to learn β€” just functions and a list, which means the whole mechanism fits in one screen and any Python developer can debug it. When the workflow genuinely becomes a graph rather than a line, that is the moment to look at a real orchestrator; until then, the list is both simpler and easier to reason about. The complete pipeline workflow shows how the same idea scales up to extract-and-load stages either side of these transforms.

Steps that break the chain

Deciding whether a piece of work belongs in the chain or at one end of it. what belongs in the chainWhat does this piece of work do?reads or writesa bookendoutside the chainreprojectsfirst, or assertit moves an invariantframe in, frame outa stepobeys the contractLog the feature count at every step: a filter that drops everything is otherwise silent spatialworkflow.io
Forcing extract and load into the step contract produces steps that lie about their return values.

A step that must not run in the middle

Reading the source and writing the output are not transform steps β€” they do not obey the contract, and forcing them into it produces steps that lie about their return values. Keep extract and load as bookends around the chain, as in the complete pipeline workflow.

Steps that change the CRS

Any step that reprojects changes an invariant every later step depends on. Either put reprojection first and treat the CRS as fixed for the rest of the chain, or have measurement steps assert the CRS they need: assert gdf.crs.is_projected, "reproject before measuring". Silent CRS drift is the most common cause of results that look plausible and are wrong.

Steps that drop every feature

A filter with a wrong threshold returns an empty frame, and every later step happily processes zero rows. Log feature counts at every step β€” as the runner above does β€” and the collapse is obvious in the log rather than invisible in the output.

Mutating the input by accident

gdf["col"] = … modifies the frame in place, so a step without a .copy() changes its caller's data. That breaks the mental model of a chain and produces bugs that move when you reorder steps. Copy at the top of any step that assigns.

Performance of copying at every step

Copying a large GeoDataFrame in every step costs memory and time. For most workflows this is irrelevant; for large data, copy once at the start of the chain and let the steps mutate deliberately, or drop columns early so what you copy is small. Measure before optimising β€” the reprojection is almost always the expensive part, not the copies. See speed up GeoPandas for large datasets.

Order-dependent steps

Some pairs genuinely cannot be reordered: you must repair geometries before dissolving, and reproject before measuring area. Encode those dependencies as assertions inside the steps rather than as comments, so a wrong order fails immediately with a clear message instead of producing quiet nonsense.

FAQ

What should the signature of a pipeline step be?

(gdf, cfg) -> gdf. The GeoDataFrame is the data flowing through; the config carries settings and any extra layers the step needs. Keeping every step to that one signature is what makes them interchangeable β€” and it costs nothing, because a step that ignores cfg simply does not use it.

How do I pass a parameter to a single step?

Use a factory: a function that takes the parameter and returns the step, closed over it. buffer(25) returns a step that buffers by 25 metres. This keeps the parameter visible in the pipeline list and lets the same step appear twice with different settings β€” impossible with a single config key.

Should steps write files?

No. A step that writes to disk cannot be reordered, cannot be tested without a filesystem, and hides an effect the pipeline list does not show. Keep reading and writing in dedicated extract and load stages at either end of the chain, and let every step in the middle be a pure transform.

How do I skip a step conditionally?

Build the list conditionally rather than putting an if inside the step: steps = [drop_empty, reproject] + ([clip] if cfg.get("boundary") else []) + [add_area]. The pipeline still reads as a list, and the condition is visible where the workflow is described instead of buried in a function.

Can I run steps in parallel?

Steps in a chain are sequential by definition β€” each needs the previous one's output. Parallelism belongs inside a step, across features or files: split the frame, map the work over a pool, concatenate the results. See speed up batch GIS jobs with parallel processing for the pattern and its caveats.

How is this different from just calling the functions in order?

Mechanically, very little β€” and that is the point. What changes is that the workflow becomes a value you can name, store, reorder, log, and read from a config file, instead of control flow buried in a script. The fold gives you one place to add logging, timing, or validation for every step at once.

Where does error handling go?

In the runner, not the steps. Wrap the loop so a failure is reported with the step's name attached, and let steps raise plain exceptions describing what went wrong. That keeps each step readable and gives every failure a consistent, useful message. For flaky steps that deserve another attempt, see add retries and timeouts.