How to Cache Pipeline Steps So Only Changed Data Is Reprocessed

There is a particular kind of waste that only shows up once a pipeline is real: you change one line in the final reporting stage, rerun, and sit through the same slow stages again — on this page, a minute and a half clipping the Isle of Wight's 71,707 buildings to its coastline — to produce exactly the intermediate result they produced an hour ago. The data has not changed. The parameters have not changed. The code for those stages has not changed. Nothing about the work is new, and you pay for all of it. Caching fixes that — but only if the cache key is honest about everything a stage depends on, because a cache that returns a stale result is far worse than no cache at all.

Problem statement

Reruns cost far more than the change that triggered them. The symptoms:

• Every rerun starts from zero — even when only the last stage changed.
• Development is slow — iterating on the final map means waiting for the slow stages every time.
• Hand-rolled skipping is unsafe — if not out.exists(): compute() silently reuses output produced with different parameters.
• A whole folder reprocessed for two new files — a nightly batch redoes every file because two arrived.
• No idea what is stale — you cannot tell which cached artefacts are still trustworthy, so you delete everything to be safe.

A stage should compute once for each combination of inputs, parameters and code, reuse that result while nothing relevant changes, and recompute by itself when something does.

Quick answer

Build a cache key by hashing everything the stage depends on — the input file's content or its size and modification time, the parameters, and the version of the step — then store the result under that key. If the key exists, load it; otherwise compute and save.

import hashlib, json
from pathlib import Path
import geopandas as gpd

CACHE = Path(".cache")
CACHE.mkdir(exist_ok=True)

def fingerprint(path: Path) -> str:
    """Cheap file identity: size + modification time."""
    st = path.stat()
    return f"{path.name}:{st.st_size}:{int(st.st_mtime)}"

def cache_key(step_name: str, version: str, inputs, params: dict) -> str:
    payload = {
        "step": step_name,
        "version": version,
        "inputs": sorted(fingerprint(Path(p)) for p in inputs),
        "params": params,
    }
    blob = json.dumps(payload, sort_keys=True, default=str)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

def cached_step(step_name, version, inputs, params, compute):
    key = cache_key(step_name, version, inputs, params)
    path = CACHE / f"{step_name}-{key}.gpkg"
    if path.exists():
        print(f"cache hit  {step_name} [{key}]")
        return gpd.read_file(path)
    print(f"cache miss {step_name} [{key}] — computing")
    result = compute()
    tmp = path.with_suffix(".gpkg.tmp")
    result.to_file(tmp, driver="GPKG")
    tmp.replace(path)
    return result

import time

for run in (1, 2):
    started = time.perf_counter()
    buildings = cached_step(
        "reproject_clip", version="3",
        inputs=[cfg["source"], cfg["boundary"]],
        params={"crs": cfg["target_crs"], "min_area": cfg["min_area_m2"]},
        compute=lambda: clip_and_reproject(cfg),
    )
    print(f"run {run}: {len(buildings):,} buildings in {time.perf_counter() - started:.2f} s")

cache miss reproject_clip [1b1d5e1328c3e213] — computing
run 1: 70,032 buildings in 91.95 s
cache hit  reproject_clip [1b1d5e1328c3e213]
run 2: 70,032 buildings in 0.12 s

Key, fingerprint, version, store

Decide what actually goes into the key

A cache key must include everything that could change the result and nothing that cannot. Miss something and you will serve a stale answer; include noise like a timestamp and you will never get a hit.

Include: input file identity, every parameter the step reads, and a version string you bump when you change the step's logic. Exclude: wall-clock time, the output path, log levels, and anything cosmetic.

payload = {
    "step": "clip_to_boundary",
    "version": "3",                                  # bump when the code changes
    "inputs": ["parcels.gpkg:184320:1785612345",
               "district.geojson:9210:1785600000"],
    "params": {"crs": "EPSG:27700", "predicate": "intersects"},
}

Choose how to fingerprint inputs

Two options, with a clear trade-off. Size plus modification time costs microseconds but is fooled by a file that was rewritten with identical content, or by a copy that reset the timestamp. A content hash is exact but reads the whole file — a real cost at gigabyte scale, though still far cheaper than a reprojection.

def content_hash(path: Path, chunk=1 << 20) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for block in iter(lambda: fh.read(chunk), b""):
            h.update(block)
    return h.hexdigest()[:16]

def fingerprint(path: Path, mode="stat") -> str:
    return content_hash(path) if mode == "hash" else f"{path.stat().st_size}:{int(path.stat().st_mtime)}"

The stat fingerprint has a sharper edge than "fooled by a rewrite". A GeoPackage is a SQLite database whose size moves in whole pages, so an edit that changes a value rather than adding rows usually leaves the size exactly as it was — and a copy or sync that preserves timestamps (cp -p, rsync -t, most backup restores) puts the old modification time back:

import os
import shutil
import timeit

shutil.copy2(cfg["source"], "data/raw/edited.gpkg")
edited = Path("data/raw/edited.gpkg")
before = (fingerprint(edited, "stat"), fingerprint(edited, "hash"))
stamp = edited.stat()
frame = gpd.read_file(edited)                               # a real edit: retag one building
frame.loc[0, "building"] = "demolished"
frame.to_file(edited, layer="buildings")
print("size unchanged by the edit:", edited.stat().st_size == stamp.st_size)
os.utime(edited, ns=(stamp.st_atime_ns, stamp.st_mtime_ns))  # what rsync -t does on arrival
after = (fingerprint(edited, "stat"), fingerprint(edited, "hash"))
print("stat fingerprint unchanged:", before[0] == after[0], before[0])
print("content hash unchanged:    ", before[1] == after[1])
print(f"hashing {edited.stat().st_size / 1e6:.1f} MB took "
      f"{min(timeit.repeat(lambda: content_hash(edited), number=1, repeat=3)) * 1000:.0f} ms")

size unchanged by the edit: True
stat fingerprint unchanged: True 18272256:1791667416
content hash unchanged:     False
hashing 18.3 MB took 10 ms

The edit is invisible to the stat fingerprint, so a cached result built from the old file would be served as current. The int() around st_mtime opens a second hole of the same kind — two writes within one second look identical — which st_mtime_ns closes; nothing about timestamps closes the first. The content hash sees both, and on this 18.3 MB file it took 10 ms, against a stage of a minute and a half.

Use stat during development where speed matters, and hash for scheduled production runs where correctness matters more. Make it a config setting rather than a code edit.

Version the step, not just the data

The most common stale-cache bug is code that changed while the inputs did not. Nothing about the files tells the cache that your clip now uses a different predicate. A version string you bump by hand is crude and works; hashing the function's source is automatic and catches what you forget.

import inspect, hashlib

def source_version(fn) -> str:
    src = inspect.getsource(fn)
    return hashlib.sha256(src.encode()).hexdigest()[:8]

print("clip_and_reproject", source_version(clip_and_reproject))

clip_and_reproject 3721ec11

Source hashing has a limit worth knowing: it sees the step function, not the helpers it calls. If a step delegates to a module you edit often, include that module's version too, or bump manually.

Store the artefact and its metadata together

A cache entry that cannot explain itself is impossible to debug. Write a small JSON sidecar next to each artefact recording what produced it.

def save_entry(path: Path, result, meta: dict):
    tmp = path.with_suffix(path.suffix + ".tmp")
    result.to_file(tmp, driver="GPKG")
    tmp.replace(path)
    path.with_suffix(".json").write_text(json.dumps(meta, indent=2, default=str))

from datetime import datetime, timezone

meta = {
    "step": "reproject_clip",
    "version": "3",
    "inputs": [str(cfg["source"]), str(cfg["boundary"])],
    "params": {"crs": cfg["target_crs"], "min_area": cfg["min_area_m2"]},
    "rows": len(buildings),
    "created": datetime.now(timezone.utc).isoformat(timespec="seconds"),
}
entry = CACHE / "reproject_clip-annotated.gpkg"
save_entry(entry, buildings, meta)
print(sorted(p.name for p in CACHE.glob("reproject_clip-annotated*")))

['reproject_clip-annotated.gpkg', 'reproject_clip-annotated.json']

Now .cache/*.json answers "what is in here and can I trust it?" without opening a single GeoPackage.

Make the cache a decorator so steps stay clean

Caching should not leak into your GIS logic. A decorator keeps the step a pure function of its inputs.

import functools

def disk_cache(version="1", cache_dir=CACHE, key_params=()):
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(inputs, params, *args, **kwargs):
            relevant = {k: params[k] for k in key_params} if key_params else params
            key = cache_key(fn.__name__, version, inputs, relevant)
            path = cache_dir / f"{fn.__name__}-{key}.gpkg"
            if path.exists():
                return gpd.read_file(path)
            result = fn(inputs, params, *args, **kwargs)
            save_entry(path, result, {"step": fn.__name__, "key": key,
                                      "rows": len(result)})
            return result
        return wrapper
    return decorator

@disk_cache(version="3", key_params=("target_crs", "min_area_m2"))
def reproject_and_filter(inputs, params):
    gdf = gpd.read_file(inputs[0]).to_crs(params["target_crs"])
    gdf["area_m2"] = gdf.geometry.area
    return gdf[gdf["area_m2"] >= params["min_area_m2"]]

before = set(CACHE.glob("reproject_and_filter-*.gpkg"))
for params in (cfg, cfg, {**cfg, "report_title": "changed"}, {**cfg, "min_area_m2": 50}):
    rows = len(reproject_and_filter([cfg["source"]], params))
    made = set(CACHE.glob("reproject_and_filter-*.gpkg")) - before
    before |= made
    print(f"{rows:6,} rows, {'computed' if made else 'from cache'}")

70,079 rows, computed
70,079 rows, from cache
70,079 rows, from cache
50,782 rows, computed

key_params is a useful discipline: it forces you to state which parameters the step actually depends on, so an unrelated config change does not invalidate everything.

Give every run a way to bypass the cache

Trust in a cache comes from being able to ignore it. Two flags are enough: one to skip reads, one to clear entries.

p.add_argument("--no-cache", action="store_true", help="recompute everything")
p.add_argument("--clear-cache", action="store_true", help="delete cached artefacts first")

if args.clear_cache:
    for f in CACHE.glob("*"):
        f.unlink()
if args.no_cache:
    os.environ["PIPELINE_CACHE"] = "off"

Prune it, or it will fill the disk

A cache keyed on content grows a new entry every time the input changes. Delete by age and cap the total size.

import time

def prune(cache_dir=CACHE, max_age_days=30, max_gb=20):
    now = time.time()
    files = sorted(cache_dir.glob("*.gpkg"), key=lambda p: p.stat().st_mtime, reverse=True)
    total = 0
    for f in files:
        age_days = (now - f.stat().st_mtime) / 86400
        total += f.stat().st_size
        if age_days > max_age_days or total > max_gb * 1e9:
            f.unlink(missing_ok=True)
            f.with_suffix(".json").unlink(missing_ok=True)

sizes = {f.name: f"{f.stat().st_size / 1e6:.1f} MB" for f in CACHE.glob("*.gpkg")}
prune()
print(len(sizes), "entries before,", len(list(CACHE.glob("*.gpkg"))), "after:", sizes)

4 entries before, 4 after: {'reproject_clip-annotated.gpkg': '18.5 MB', 'reproject_and_filter-462b64474cc199ca.gpkg': '18.8 MB', 'reproject_clip-1b1d5e1328c3e213.gpkg': '18.5 MB', 'reproject_and_filter-32c286b412a76db3.gpkg': '14.0 MB'}

Caching steps over the Isle of Wight's buildings

The data used on this page

The Isle of Wight from OpenStreetMap (Geofabrik's extract, ODbL): its 71,707 building footprints and its land-use polygons, read from the raw .osm.pbf, and the island's full-resolution boundary from the ONS Local Authority Districts (May 2024). Clipping every building to that coastline is the slow, stable stage a cache exists for. The cache folder is emptied first, so every run of this page starts cold.

import shutil
from pathlib import Path

import geopandas as gpd
import requests

Path("data/raw").mkdir(parents=True, exist_ok=True)
PBF = Path("isle-of-wight-260101.osm.pbf")
if not PBF.exists():
    PBF.write_bytes(requests.get("https://download.geofabrik.de/europe/united-kingdom/england/"
                                 "isle-of-wight-260101.osm.pbf", timeout=300).content)
if not Path("data/raw/buildings.gpkg").exists():
    gpd.read_file(PBF, layer="multipolygons", where="building IS NOT NULL",
                  columns=["osm_id", "osm_way_id", "building"]).to_file("data/raw/buildings.gpkg", layer="buildings")
    gpd.read_file(PBF, layer="multipolygons", where="landuse IS NOT NULL",
                  columns=["osm_id", "landuse"]).to_file("data/raw/landuse.gpkg")
    gpd.read_file("https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/"
                  "Local_Authority_Districts_May_2024_Boundaries_UK_BFC/FeatureServer/0/query"
                  "?where=LAD24CD%3D%27E06000046%27&outFields=LAD24CD,LAD24NM&outSR=27700&f=geojson"
                  ).to_file("data/raw/boundary.gpkg")
shutil.rmtree(".cache", ignore_errors=True)

cfg = {"source": Path("data/raw/buildings.gpkg"), "boundary": Path("data/raw/boundary.gpkg"),
       "target_crs": "EPSG:27700", "min_area_m2": 10}

def clip_and_reproject(cfg):
    """The slow stage: every building, clipped to the island's coastline."""
    buildings = gpd.read_file(cfg["source"]).to_crs(cfg["target_crs"])
    clipped = gpd.clip(buildings, gpd.read_file(cfg["boundary"]).to_crs(cfg["target_crs"]))
    return clipped[clipped.geometry.area >= cfg["min_area_m2"]]

print(len(gpd.read_file(cfg["source"])), "buildings;",
      f"{cfg['source'].stat().st_size / 1e6:.1f} MB;",
      gpd.read_file(cfg["boundary"]).LAD24NM.item())

71005 buildings; 18.3 MB; Isle of Wight

Example 1: Skipping files that have not changed in a batch job

The most valuable caching in GIS is often the simplest: in a folder job, skip a file whose output is newer than its input.

def needs_rebuild(src: Path, out: Path) -> bool:
    if not out.exists():
        return True
    return src.stat().st_mtime > out.stat().st_mtime

import os

src_dir, out_dir = Path("data/by_landuse"), Path("data/out")
src_dir.mkdir(exist_ok=True); out_dir.mkdir(exist_ok=True)
landuse = gpd.read_file("data/raw/landuse.gpkg")
for kind in ("residential", "farmland", "meadow", "forest", "grass", "industrial"):
    landuse[landuse.landuse == kind].to_file(src_dir / f"{kind}.gpkg")

def process_file(src, out):
    gpd.read_file(src).to_crs(27700).to_file(out)

def run_folder():
    processed = skipped = 0
    for src in sorted(src_dir.glob("*.gpkg")):
        out = out_dir / f"{src.stem}.gpkg"
        if not needs_rebuild(src, out):
            skipped += 1
            continue
        process_file(src, out)
        processed += 1
    print(f"{processed} processed, {skipped} up to date")

run_folder()
run_folder()
os.utime(src_dir / "meadow.gpkg")                  # one file arrives again
run_folder()

6 processed, 0 up to date
0 processed, 6 up to date
1 processed, 5 up to date

This is exactly what make does, and for a nightly job over a slowly changing folder it turns a full run into a few seconds. Combine it with batch processing a folder.

Example 2: In-memory memoisation within one run

When a stage is called repeatedly with the same arguments inside a single run, disk is unnecessary.

from functools import lru_cache

@lru_cache(maxsize=8)
def load_boundary(path: str, crs: str):
    return gpd.read_file(path).to_crs(crs)

tiles = [gpd.read_file(f) for f in sorted(out_dir.glob("*.gpkg"))]
for tile in tiles:                                  # read once, reused by every tile
    boundary = load_boundary(str(cfg["boundary"]), cfg["target_crs"])
    gpd.clip(tile, boundary)
print(load_boundary.cache_info())

boundary = load_boundary(str(cfg["boundary"]), cfg["target_crs"])
boundary["geometry"] = boundary.buffer(-1000)     # one caller trims the coast for its own use
print(f"{load_boundary(str(cfg['boundary']), cfg['target_crs']).area.item() / 1e6:.1f} km² "
      f"now comes back from the cache, against {gpd.read_file(cfg['boundary']).area.item() / 1e6:.1f} km² on disk")

CacheInfo(hits=5, misses=1, maxsize=8, currsize=1)
252.9 km² now comes back from the cache, against 379.6 km² on disk

lru_cache needs hashable arguments — pass strings and paths, never GeoDataFrames. And it hands every caller the same object: the one caller that modified its copy changed what every later call receives, for the rest of the run. Return gdf.copy() from a memoised loader, or treat what it returns as read-only.

Example 3: Caching a slow remote fetch

Network reads are the best caching candidates: slow, rate-limited, and usually unchanged between runs.

from datetime import datetime, timedelta, timezone

def fetch_cached(url, params, max_age=timedelta(hours=12)):
    key = hashlib.sha256(f"{url}{sorted(params.items())}".encode()).hexdigest()[:16]
    path = CACHE / f"wfs-{key}.gpkg"
    if path.exists():
        age = datetime.now(timezone.utc) - datetime.fromtimestamp(
            path.stat().st_mtime, timezone.utc)
        if age < max_age:
            return gpd.read_file(path)
    gdf = gpd.read_file(url, **params)
    gdf.to_file(path, driver="GPKG")
    return gdf

ONS = ("https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/"
       "Local_Authority_Districts_May_2024_Boundaries_UK_BGC/FeatureServer/0/query"
       "?where=LAD24CD%3D%27E06000046%27&outFields=LAD24NM&outSR=27700&f=geojson")
for attempt in (1, 2):
    started = time.perf_counter()
    lad = fetch_cached(ONS, {})
    print(f"call {attempt}: {lad.LAD24NM.item()} in {time.perf_counter() - started:.2f} s")

call 1: Isle of Wight in 0.56 s
call 2: Isle of Wight in 0.00 s

A time-to-live is the right invalidation strategy here, because you cannot fingerprint a remote dataset without downloading it.

Example 4: A cached pipeline runner

Give the runner the cache and every stage in every pipeline benefits at once.

def run_cached(gdf_source, steps, cfg, log):
    inputs = [cfg["source"]]
    data = None
    for step in steps:
        version = getattr(step, "cache_version", "1")
        key = cache_key(step.__name__, version, inputs, cfg["params"])
        path = CACHE / f"{step.__name__}-{key}.gpkg"

        if path.exists() and cfg.get("use_cache", True):
            log.info("%-24s cache hit  [%s]", step.__name__, key)
            data = gpd.read_file(path)
            continue

        t0 = time.perf_counter()
        data = step(data if data is not None else gpd.read_file(gdf_source), cfg)
        log.info("%-24s computed   [%s] in %.1fs",
                 step.__name__, key, time.perf_counter() - t0)
        save_entry(path, data, {"step": step.__name__, "rows": len(data)})
    return data

Two real steps — the slow clip, then a land-use tag from a spatial join — run through it three times: cold, warm, and after the clip is changed to drop buildings under 50 m² and its version bumped:

import logging
import sys

class Stdout(logging.Handler):
    def emit(self, record):
        print(self.format(record))

log = logging.getLogger("pipeline")
log.setLevel(logging.INFO)
log.addHandler(Stdout())
log.propagate = False                              # keep GDAL's own INFO chatter out of it
landuse_27700 = gpd.read_file("data/raw/landuse.gpkg").to_crs(27700)

def clip_to_island(data, cfg):
    clipped = gpd.clip(data.to_crs(27700), gpd.read_file(cfg["boundary"]))
    return clipped[clipped.area >= cfg["min_area"]]
clip_to_island.cache_version = "1"

def tag_landuse(data, cfg):
    joined = gpd.sjoin(data, landuse_27700[["landuse", "geometry"]], how="left", predicate="intersects")
    return joined[~joined.index.duplicated()].drop(columns="index_right")

pcfg = {"source": cfg["source"], "boundary": cfg["boundary"], "min_area": 10,
        "params": {"crs": "EPSG:27700"}}
for label in ("cold", "warm"):
    out = run_cached(cfg["source"], [clip_to_island, tag_landuse], pcfg, log)
    print(f"-> {label}: {len(out):,} tagged buildings\n")

pcfg["min_area"] = 50                              # the clip now drops small sheds…
clip_to_island.cache_version = "2"                 # …and its version is bumped, as it should be
out = run_cached(cfg["source"], [clip_to_island, tag_landuse], pcfg, log)
print(f"-> after the change: {len(out):,} tagged buildings, "
      f"{(out.area < 50).sum():,} of them under 50 m²")

clip_to_island           computed   [c72a22354b7b2e97] in 93.1s
tag_landuse              computed   [bca18d06fcad8a13] in 0.4s
-> cold: 70,032 tagged buildings

clip_to_island           cache hit  [c72a22354b7b2e97]
tag_landuse              cache hit  [bca18d06fcad8a13]
-> warm: 70,032 tagged buildings

clip_to_island           computed   [fe9e552f886e53dc] in 93.2s
tag_landuse              cache hit  [bca18d06fcad8a13]
-> after the change: 70,032 tagged buildings, 19,298 of them under 50 m²

The clip recomputed, as it should. The land-use step did not: its key is built from its own name, its own version and the original source file, none of which changed, so it served the output it had built from the old clip — thousands of buildings the new clip had removed. The key of a later stage has to include the key of the stage before it:

def run_cached_chained(gdf_source, steps, cfg, log):
    data, previous = None, None
    for step in steps:
        version = getattr(step, "cache_version", "1")
        params = {**cfg["params"], **({"upstream": previous} if previous else {})}
        key = cache_key(step.__name__, version, [cfg["source"]], params)   # the fix
        path = CACHE / f"{step.__name__}-{key}.gpkg"
        if path.exists() and cfg.get("use_cache", True):
            log.info("%-24s cache hit  [%s]", step.__name__, key)
            data = gpd.read_file(path)
        else:
            t0 = time.perf_counter()
            data = step(data if data is not None else gpd.read_file(gdf_source), cfg)
            log.info("%-24s computed   [%s] in %.1fs", step.__name__, key, time.perf_counter() - t0)
            save_entry(path, data, {"step": step.__name__, "rows": len(data)})
        previous = key
    return data

out = run_cached_chained(cfg["source"], [clip_to_island, tag_landuse], pcfg, log)
print(f"-> chained: {len(out):,} tagged buildings, {(out.area < 50).sum():,} under 50 m²")

clip_to_island           cache hit  [fe9e552f886e53dc]
tag_landuse              computed   [b8a38dcb53b3509a] in 0.3s
-> chained: 50,734 tagged buildings, 0 under 50 m²

The first step's key is unchanged, because it has no upstream, so the version-2 clip comes straight from the cache; the tag step's key now depends on it, so it recomputes against the right input.

Example 5: Inspecting the cache

A three-line report makes the cache visible instead of magical.

import pandas as pd

def cache_report(cache_dir=CACHE):
    rows = []
    for meta_path in sorted(cache_dir.glob("*.json")):
        meta = json.loads(meta_path.read_text())
        data_path = meta_path.with_suffix(".gpkg")
        rows.append({
            "step": meta.get("step"),
            "rows": meta.get("rows"),
            "MB": round(data_path.stat().st_size / 1e6, 1) if data_path.exists() else 0,
            "created": meta.get("created", "?"),
        })
    return pd.DataFrame(rows).sort_values("MB", ascending=False)

print(cache_report().to_string(index=False))

                step  rows   MB                   created
         tag_landuse 70032 19.3                         ?
reproject_and_filter 70079 18.8                         ?
      clip_to_island 70032 18.5                         ?
      reproject_clip 70032 18.5 2026-10-10T21:25:11+00:00
         tag_landuse 50734 14.4                         ?
reproject_and_filter 50782 14.0                         ?
      clip_to_island 50734 13.8                         ?

Caching trades correctness risk for time

Caching is a trade: correctness risk for time. The time saved is obvious and immediate; the risk is that a key omits something the result depended on, and the pipeline silently serves an answer computed under conditions that no longer hold. That is why the key deserves more design thought than the storage. Every field in the key is a claim — "if these are the same, the result is the same" — and each claim should be one you would defend.

The code-version problem is the one that catches everyone. File-based invalidation feels complete because it covers the data, and it silently misses the case where you edited the transform. A pipeline that caches on inputs alone will confidently hand you yesterday's logic applied to today's data. Bump a version string, hash the source, or — most robustly for a team — include the git commit of the working tree in the key and accept that an uncommitted change means no cache hits.

Where to cache matters as much as how. Caching a fast step wastes disk and adds a serialisation round-trip that can be slower than recomputing. The candidates worth caching are the ones that are slow and stable: a reprojection of a large layer, a spatial join across two big datasets, a remote download, a raster resample. Time your stages first — the runner in the complete pipeline workflow prints per-stage timings for exactly this reason — and cache the top two.

There is also a format cost people forget. Round-tripping a GeoDataFrame through GeoPackage is not free, and it is not perfectly lossless:

import numpy as np
import pandas as pd

sample = buildings.head(1000).copy()
sample["levels"] = pd.array(np.where(np.arange(1000) % 3, 2, None), dtype="Int64")   # a nullable integer
sample["surveyed"] = pd.Timestamp("2026-09-01 09:30", tz="Europe/London")
sample["listed"] = np.arange(1000) % 7 == 0
sample.to_file("roundtrip.gpkg"); sample.to_parquet("roundtrip.parquet")
for name, back in (("GeoPackage", gpd.read_file("roundtrip.gpkg")),
                   ("GeoParquet", gpd.read_parquet("roundtrip.parquet"))):
    print(f"{name:11}", {c: str(back[c].dtype) for c in ("levels", "surveyed", "listed")})

buildings.to_file("all.gpkg"); buildings.to_parquet("all.parquet")
for name, read in (("GeoPackage", lambda: gpd.read_file("all.gpkg")),
                   ("GeoParquet", lambda: gpd.read_parquet("all.parquet"))):
    print(f"{name:11} reads {len(buildings):,} buildings in "
          f"{min(timeit.repeat(read, number=1, repeat=3)):.3f} s")

GeoPackage  {'levels': 'float64', 'surveyed': 'datetime64[ms, UTC+01:00]', 'listed': 'bool'}
GeoParquet  {'levels': 'Int64', 'surveyed': 'datetime64[us, Europe/London]', 'listed': 'bool'}
GeoPackage  reads 70,032 buildings in 0.119 s
GeoParquet  reads 70,032 buildings in 0.047 s

For intermediate artefacts within one machine, GeoParquet keeps the dtypes and reads faster. Save GeoPackage for the artefacts a human will open in QGIS.

Finally, a cache must be disposable. rm -rf .cache should never be frightening — worst case, the next run is slow. Keep the cache out of version control, never make it the only copy of anything, and treat "delete it and rerun" as the first debugging step whenever a result looks impossible.

Modification times that lie, and other stale keys

Modification times lie

Copying files, restoring a backup, or checking out a git branch rewrites mtime without changing content — producing spurious misses — while some sync tools preserve mtime while changing content, producing dangerous hits. When correctness matters more than milliseconds, hash the content.

The cache is not a store of record

Nothing you cannot regenerate belongs in a cache. If an artefact is genuinely the output of a run, write it to the output folder with a manifest, as in making a workflow reproducible. Caches get pruned, deleted, and invalidated by design.

Two runs writing the same key

A parallel run or two scheduled jobs can compute the same key simultaneously and write the same file. Always write to a temporary path and replace() into place — an atomic rename means the worst outcome is duplicated work rather than a corrupt artefact.

Floating-point parameters in keys

json.dumps({"tolerance": 0.1 + 0.2}) produces 0.30000000000000004, which will never match a later 0.3. Round numeric parameters to a sensible precision before they go into the key.

Caching non-deterministic steps

A step that samples randomly, calls datetime.now(), or depends on a remote service's current state is not a function of its declared inputs, so a cached result is arbitrary rather than correct. Either make it deterministic (pass a seed and put it in the key) or exclude it from caching.

Cache size on raster work

Raster intermediates are enormous, and a naïve cache will fill a disk quickly — which then fails every job on that machine. Set a hard size cap with pruning, keep the cache on a volume that is not the system disk, and consider caching only the metadata for raster steps where the reads are cheap.

Internal links

• How to Build a GIS Data Pipeline in Python
• Chain GIS Processing Steps into a Reusable Pipeline
• Schedule a Python GIS Script to Run Automatically
• Make a GIS Workflow Reproducible
• Build a Resumable Batch GIS Job in Python
• Batch Process a Folder of GIS Files in Python
• How to Speed Up GeoPandas: Tips for Large Datasets

FAQ

What should go into a cache key?

Everything that can change the result: a fingerprint of each input file, every parameter the step reads, and a version identifying the step's code. Nothing that cannot: timestamps, output paths, log levels. A missing field produces stale results; an irrelevant one produces permanent cache misses.

Should I hash file contents or use modification times?

Modification time and size are effectively free and good enough while developing. Content hashes are exact and worth the read for scheduled production runs, where a restored backup or a git checkout could otherwise change mtime without changing data — or, worse, change data without changing mtime. Make it a config option.

How does the cache know my code changed?

It does not, unless you tell it. Include a version string in the key and bump it when you change the step, or hash the function's source with inspect.getsource. Source hashing catches what you forget but only sees that function, not the helpers it calls — for a team, hashing the git commit is the most reliable option.

Which steps are worth caching?

Slow and stable ones: large reprojections, spatial joins over big datasets, remote downloads, raster resampling. Caching a fast step costs disk and a serialisation round-trip that may exceed the computation. Time the stages first and cache the two slowest.

What format should cached intermediates use?

GeoParquet or Feather for machine-to-machine intermediates — they preserve dtypes better and read much faster than GeoPackage. Use GeoPackage when a human may open the artefact in QGIS. Whatever the format, write to a temporary file and rename it into place so a partially written artefact is never readable.

How is this different from a resumable batch job?

Caching asks "has this exact work been done before?" and applies across runs and across time. Resumability asks "where did this run stop?" and applies within one job over many files. They compose well — see build a resumable batch GIS job — but solve different problems.

Is it safe to delete the cache?

Always. If deleting it can lose something, it was not a cache — it was an output stored in the wrong place. Keep .cache/ out of version control, prune it on a schedule, and treat "clear it and rerun" as the first step whenever a result looks wrong.