⚙️ How-To

How to Cache Pipeline Steps So Only Changed Data Is Reprocessed

There is a particular kind of waste that only shows up once a pipeline is real: you change one line in the final reporting stage, rerun, and sit through eighteen minutes of reprojection and spatial joins that produce byte-for-byte the same intermediate result they produced an hour ago. The data has not changed. The parameters have not changed. The code for those stages has not changed. Nothing about the work is new, and you pay for all of it. Caching fixes that — but only if the cache key is honest about everything a stage depends on, because a cache that returns a stale result is far worse than no cache at all.

Problem statement

Reruns cost far more than the change that triggered them. The symptoms:

  • Every rerun starts from zero — even when only the last stage changed.
  • Development is slow — iterating on the final map means waiting for the slow stages every time.
  • Hand-rolled skipping is unsafe — if not out.exists(): compute() silently reuses output produced with different parameters.
  • A whole folder reprocessed for two new files — a nightly batch redoes 500 files because two arrived.
  • No idea what is stale — you cannot tell which cached artefacts are still trustworthy, so you delete everything to be safe.

The goal: each stage computes once per unique combination of inputs, parameters, and code, reuses its result when nothing relevant has changed, and invalidates automatically when something has.

Quick answer

Build a cache key by hashing everything the stage depends on — the input file's content or its size and modification time, the parameters, and the version of the step — then store the result under that key. If the key exists, load it; otherwise compute and save.

Input file fingerprint, parameters, and step version hashed together into a single cache key.
Everything the result depends on goes into the key — anything you leave out becomes a stale-cache bug.
import hashlib, json
from pathlib import Path
import geopandas as gpd

CACHE = Path(".cache")
CACHE.mkdir(exist_ok=True)

def fingerprint(path: Path) -> str:
    """Cheap file identity: size + modification time."""
    st = path.stat()
    return f"{path.name}:{st.st_size}:{int(st.st_mtime)}"

def cache_key(step_name: str, version: str, inputs, params: dict) -> str:
    payload = {
        "step": step_name,
        "version": version,
        "inputs": sorted(fingerprint(Path(p)) for p in inputs),
        "params": params,
    }
    blob = json.dumps(payload, sort_keys=True, default=str)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

def cached_step(step_name, version, inputs, params, compute):
    key = cache_key(step_name, version, inputs, params)
    path = CACHE / f"{step_name}-{key}.gpkg"
    if path.exists():
        print(f"cache hit  {step_name} [{key}]")
        return gpd.read_file(path)
    print(f"cache miss {step_name} [{key}] — computing")
    result = compute()
    tmp = path.with_suffix(".gpkg.tmp")
    result.to_file(tmp, driver="GPKG")
    tmp.replace(path)
    return result
parcels = cached_step(
    "reproject_clip", version="3",
    inputs=[cfg["source"], cfg["boundary"]],
    params={"crs": cfg["target_crs"], "min_area": cfg["min_area_m2"]},
    compute=lambda: clip_and_reproject(cfg),
)

Step-by-step solution

Decide what actually goes into the key

A cache key must include everything that could change the result and nothing that cannot. Miss something and you will serve a stale answer; include noise like a timestamp and you will never get a hit.

A decision: key found means load from cache, key absent means compute and store.
The whole mechanism is one lookup — the difficulty is entirely in choosing the key.

Include: input file identity, every parameter the step reads, and a version string you bump when you change the step's logic. Exclude: wall-clock time, the output path, log levels, and anything cosmetic.

payload = {
    "step": "clip_to_boundary",
    "version": "3",                                  # bump when the code changes
    "inputs": ["parcels.gpkg:184320:1785612345",
               "district.geojson:9210:1785600000"],
    "params": {"crs": "EPSG:27700", "predicate": "intersects"},
}

Choose how to fingerprint inputs

Two options, with a clear trade-off. Size plus modification time costs microseconds but is fooled by a file that was rewritten with identical content, or by a copy that reset the timestamp. A content hash is exact but reads the whole file — a real cost at gigabyte scale, though still far cheaper than a reprojection.

def content_hash(path: Path, chunk=1 << 20) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for block in iter(lambda: fh.read(chunk), b""):
            h.update(block)
    return h.hexdigest()[:16]

def fingerprint(path: Path, mode="stat") -> str:
    return content_hash(path) if mode == "hash" else f"{path.stat().st_size}:{int(path.stat().st_mtime)}"

Use stat during development where speed matters, and hash for scheduled production runs where correctness matters more. Make it a config setting rather than a code edit.

Version the step, not just the data

The most common stale-cache bug is code that changed while the inputs did not. Nothing about the files tells the cache that your clip now uses a different predicate. A version string you bump by hand is crude and works; hashing the function's source is automatic and catches what you forget.

import inspect, hashlib

def source_version(fn) -> str:
    src = inspect.getsource(fn)
    return hashlib.sha256(src.encode()).hexdigest()[:8]

Source hashing has a limit worth knowing: it sees the step function, not the helpers it calls. If a step delegates to a module you edit often, include that module's version too, or bump manually.

Store the artefact and its metadata together

A cache entry that cannot explain itself is impossible to debug. Write a small JSON sidecar next to each artefact recording what produced it.

def save_entry(path: Path, result, meta: dict):
    tmp = path.with_suffix(path.suffix + ".tmp")
    result.to_file(tmp, driver="GPKG")
    tmp.replace(path)
    path.with_suffix(".json").write_text(json.dumps(meta, indent=2, default=str))

meta = {
    "step": "reproject_clip",
    "version": "3",
    "inputs": [str(p) for p in inputs],
    "params": params,
    "rows": len(result),
    "created": datetime.now(timezone.utc).isoformat(timespec="seconds"),
    "seconds": round(elapsed, 2),
}

Now .cache/*.json answers "what is in here and can I trust it?" without opening a single GeoPackage.

Make the cache a decorator so steps stay clean

Caching should not leak into your GIS logic. A decorator keeps the step a pure function of its inputs.

import functools

def disk_cache(version="1", cache_dir=CACHE, key_params=()):
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(inputs, params, *args, **kwargs):
            relevant = {k: params[k] for k in key_params} if key_params else params
            key = cache_key(fn.__name__, version, inputs, relevant)
            path = cache_dir / f"{fn.__name__}-{key}.gpkg"
            if path.exists():
                return gpd.read_file(path)
            result = fn(inputs, params, *args, **kwargs)
            save_entry(path, result, {"step": fn.__name__, "key": key,
                                      "rows": len(result)})
            return result
        return wrapper
    return decorator

@disk_cache(version="3", key_params=("target_crs", "min_area_m2"))
def reproject_and_filter(inputs, params):
    gdf = gpd.read_file(inputs[0]).to_crs(params["target_crs"])
    gdf["area_m2"] = gdf.geometry.area
    return gdf[gdf["area_m2"] >= params["min_area_m2"]]

key_params is a useful discipline: it forces you to state which parameters the step actually depends on, so an unrelated config change does not invalidate everything.

Give every run a way to bypass the cache

Trust in a cache comes from being able to ignore it. Two flags are enough: one to skip reads, one to clear entries.

p.add_argument("--no-cache", action="store_true", help="recompute everything")
p.add_argument("--clear-cache", action="store_true", help="delete cached artefacts first")

if args.clear_cache:
    for f in CACHE.glob("*"):
        f.unlink()
if args.no_cache:
    os.environ["PIPELINE_CACHE"] = "off"

Prune it, or it will fill the disk

A cache keyed on content grows a new entry every time the input changes. Delete by age and cap the total size.

import time

def prune(cache_dir=CACHE, max_age_days=30, max_gb=20):
    now = time.time()
    files = sorted(cache_dir.glob("*.gpkg"), key=lambda p: p.stat().st_mtime, reverse=True)
    total = 0
    for f in files:
        age_days = (now - f.stat().st_mtime) / 86400
        total += f.stat().st_size
        if age_days > max_age_days or total > max_gb * 1e9:
            f.unlink(missing_ok=True)
            f.with_suffix(".json").unlink(missing_ok=True)

Code examples

Example 1: Skipping files that have not changed in a batch job

The most valuable caching in GIS is often the simplest: in a folder job, skip a file whose output is newer than its input.

def needs_rebuild(src: Path, out: Path) -> bool:
    if not out.exists():
        return True
    return src.stat().st_mtime > out.stat().st_mtime

processed = skipped = 0
for src in sorted(src_dir.glob("*.shp")):
    out = out_dir / f"{src.stem}.gpkg"
    if not needs_rebuild(src, out):
        skipped += 1
        continue
    process_file(src, out)
    processed += 1

print(f"{processed} processed, {skipped} up to date")

This is exactly what make does, and for a nightly job over a slowly changing folder it turns a full run into a few seconds. Combine it with batch processing a folder.

Example 2: In-memory memoisation within one run

When a stage is called repeatedly with the same arguments inside a single run, disk is unnecessary.

from functools import lru_cache

@lru_cache(maxsize=8)
def load_boundary(path: str, crs: str):
    return gpd.read_file(path).to_crs(crs)

# read once, reused by every tile in the loop
for tile in tiles:
    boundary = load_boundary(str(cfg["boundary"]), cfg["target_crs"])
    clip_tile(tile, boundary)

lru_cache needs hashable arguments — pass strings and paths, never GeoDataFrames.

Example 3: Caching a slow remote fetch

Network reads are the best caching candidates: slow, rate-limited, and usually unchanged between runs.

from datetime import datetime, timedelta, timezone

def fetch_cached(url, params, max_age=timedelta(hours=12)):
    key = hashlib.sha256(f"{url}{sorted(params.items())}".encode()).hexdigest()[:16]
    path = CACHE / f"wfs-{key}.gpkg"
    if path.exists():
        age = datetime.now(timezone.utc) - datetime.fromtimestamp(
            path.stat().st_mtime, timezone.utc)
        if age < max_age:
            return gpd.read_file(path)
    gdf = gpd.read_file(url, **params)
    gdf.to_file(path, driver="GPKG")
    return gdf

A time-to-live is the right invalidation strategy here, because you cannot fingerprint a remote dataset without downloading it.

Example 4: A cached pipeline runner

Give the runner the cache and every stage in every pipeline benefits at once.

def run_cached(gdf_source, steps, cfg, log):
    inputs = [cfg["source"]]
    data = None
    for step in steps:
        version = getattr(step, "cache_version", "1")
        key = cache_key(step.__name__, version, inputs, cfg["params"])
        path = CACHE / f"{step.__name__}-{key}.gpkg"

        if path.exists() and cfg.get("use_cache", True):
            log.info("%-24s cache hit  [%s]", step.__name__, key)
            data = gpd.read_file(path)
            continue

        t0 = time.perf_counter()
        data = step(data if data is not None else gpd.read_file(gdf_source), cfg)
        log.info("%-24s computed   [%s] in %.1fs",
                 step.__name__, key, time.perf_counter() - t0)
        save_entry(path, data, {"step": step.__name__, "rows": len(data)})
    return data

Note the subtlety: once a stage is cached, the chain is cached from that point, so the key of a later stage must include the keys of the stages before it. Threading the previous key into the payload does that in one line.

Example 5: Inspecting the cache

A three-line report makes the cache visible instead of magical.

import pandas as pd

def cache_report(cache_dir=CACHE):
    rows = []
    for meta_path in sorted(cache_dir.glob("*.json")):
        meta = json.loads(meta_path.read_text())
        data_path = meta_path.with_suffix(".gpkg")
        rows.append({
            "step": meta.get("step"),
            "rows": meta.get("rows"),
            "MB": round(data_path.stat().st_size / 1e6, 1) if data_path.exists() else 0,
            "created": meta.get("created", "?"),
        })
    return pd.DataFrame(rows).sort_values("MB", ascending=False)

print(cache_report().to_string(index=False))

Explanation

Caching is a trade: correctness risk for time. The time saved is obvious and immediate; the risk is that a key omits something the result depended on, and the pipeline silently serves an answer computed under conditions that no longer hold. That is why the key deserves more design thought than the storage. Every field in the key is a claim — "if these are the same, the result is the same" — and each claim should be one you would defend.

A full rerun taking eighteen minutes versus a cached rerun taking under two, with only the changed stage recomputed.
The saving is not marginal — a cached rerun recomputes only what actually changed.

The code-version problem is the one that catches everyone. File-based invalidation feels complete because it covers the data, and it silently misses the case where you edited the transform. A pipeline that caches on inputs alone will confidently hand you yesterday's logic applied to today's data. Bump a version string, hash the source, or — most robustly for a team — include the git commit of the working tree in the key and accept that an uncommitted change means no cache hits.

Where to cache matters as much as how. Caching a fast step wastes disk and adds a serialisation round-trip that can be slower than recomputing. The candidates worth caching are the ones that are slow and stable: a reprojection of a large layer, a spatial join across two big datasets, a remote download, a raster resample. Time your stages first — the runner in the complete pipeline workflow prints per-stage timings for exactly this reason — and cache the top two.

There is also a format cost people forget. Round-tripping a GeoDataFrame through GeoPackage is not free, and it is not perfectly lossless: column dtypes can shift, and a datetime may come back subtly different. For intermediate artefacts within one machine, GeoParquet or Feather preserves dtypes better and reads considerably faster than GPKG. Save GeoPackage for the artefacts a human will open in QGIS.

Finally, a cache must be disposable. rm -rf .cache should never be frightening — worst case, the next run is slow. Keep the cache out of version control, never make it the only copy of anything, and treat "delete it and rerun" as the first debugging step whenever a result looks impossible.

Edge cases or notes

Modification times lie

Copying files, restoring a backup, or checking out a git branch rewrites mtime without changing content — producing spurious misses — while some sync tools preserve mtime while changing content, producing dangerous hits. When correctness matters more than milliseconds, hash the content.

The cache is not a store of record

Nothing you cannot regenerate belongs in a cache. If an artefact is genuinely the output of a run, write it to the output folder with a manifest, as in making a workflow reproducible. Caches get pruned, deleted, and invalidated by design.

Two runs writing the same key

A parallel run or two scheduled jobs can compute the same key simultaneously and write the same file. Always write to a temporary path and replace() into place — an atomic rename means the worst outcome is duplicated work rather than a corrupt artefact.

Floating-point parameters in keys

json.dumps({"tolerance": 0.1 + 0.2}) produces 0.30000000000000004, which will never match a later 0.3. Round numeric parameters to a sensible precision before they go into the key.

Caching non-deterministic steps

A step that samples randomly, calls datetime.now(), or depends on a remote service's current state is not a function of its declared inputs, so a cached result is arbitrary rather than correct. Either make it deterministic (pass a seed and put it in the key) or exclude it from caching.

Cache size on raster work

Raster intermediates are enormous, and a naïve cache will fill a disk quickly — which then fails every job on that machine. Set a hard size cap with pruning, keep the cache on a volume that is not the system disk, and consider caching only the metadata for raster steps where the reads are cheap.

FAQ

What should go into a cache key?

Everything that can change the result: a fingerprint of each input file, every parameter the step reads, and a version identifying the step's code. Nothing that cannot: timestamps, output paths, log levels. A missing field produces stale results; an irrelevant one produces permanent cache misses.

Should I hash file contents or use modification times?

Modification time and size are effectively free and good enough while developing. Content hashes are exact and worth the read for scheduled production runs, where a restored backup or a git checkout could otherwise change mtime without changing data — or, worse, change data without changing mtime. Make it a config option.

How does the cache know my code changed?

It does not, unless you tell it. Include a version string in the key and bump it when you change the step, or hash the function's source with inspect.getsource. Source hashing catches what you forget but only sees that function, not the helpers it calls — for a team, hashing the git commit is the most reliable option.

Which steps are worth caching?

Slow and stable ones: large reprojections, spatial joins over big datasets, remote downloads, raster resampling. Caching a fast step costs disk and a serialisation round-trip that may exceed the computation. Time the stages first and cache the two slowest.

What format should cached intermediates use?

GeoParquet or Feather for machine-to-machine intermediates — they preserve dtypes better and read much faster than GeoPackage. Use GeoPackage when a human may open the artefact in QGIS. Whatever the format, write to a temporary file and rename it into place so a partially written artefact is never readable.

How is this different from a resumable batch job?

Caching asks "has this exact work been done before?" and applies across runs and across time. Resumability asks "where did this run stop?" and applies within one job over many files. They compose well — see build a resumable batch GIS job — but solve different problems.

Is it safe to delete the cache?

Always. If deleting it can lose something, it was not a cache — it was an output stored in the wrong place. Keep .cache/ out of version control, prune it on a schedule, and treat "clear it and rerun" as the first step whenever a result looks wrong.