How to Cache Downloaded GIS Data So You Fetch It Once
Problem statement
A notebook that downloads its own data is a joy for the first ten minutes and a liability after that:
- Every rerun waits ninety seconds for the same boundaries.
- The public service you are hitting starts returning 429.
- A colleague reruns it in March and gets different data, so their figures do not match yours.
- The service goes down for maintenance and your analysis cannot run at all.
The naive fix makes things worse:
if not Path("boundaries.gpkg").exists():
download_boundaries("boundaries.gpkg")
gdf = gpd.read_file("boundaries.gpkg")That file now never updates. Six months later the analysis is running on boundaries that predate two municipal mergers, and nothing anywhere records that fact.
A cache is not "check if the file exists". A cache is a stored response plus enough metadata to decide whether it is still good.
Quick answer
Key on the request, store the payload, and record when it arrived:
import hashlib
import json
from datetime import datetime, timedelta, timezone
from pathlib import Path
import requests
CACHE = Path("data/cache")
def cached_get(url, params=None, *, max_age=timedelta(days=7)):
CACHE.mkdir(parents=True, exist_ok=True)
key = hashlib.sha256(f"{url}|{sorted((params or {}).items())}".encode()).hexdigest()[:16]
body, meta = CACHE / f"{key}.bin", CACHE / f"{key}.json"
if body.exists() and meta.exists():
info = json.loads(meta.read_text())
age = datetime.now(timezone.utc) - datetime.fromisoformat(info["retrieved"])
if age < max_age:
print(f" cache hit ({age.days}d old): {info['url'][:60]}")
return body.read_bytes()
r = requests.get(url, params=params, timeout=180)
r.raise_for_status()
body.write_bytes(r.content)
meta.write_text(json.dumps({
"url": r.url,
"retrieved": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"bytes": len(r.content),
"sha256": hashlib.sha256(r.content).hexdigest(),
}, indent=2))
print(f" fetched {len(r.content) / 1e6:.1f} MB")
return body.read_bytes()Three properties make this a cache rather than a folder: the key covers the whole request, the entry has an expiry, and the metadata records when and what.
Step-by-step solution
1. Key on everything that changes the response
The commonest cache bug is a key that is too narrow. If you key on the URL but not the parameters, a request for Manchester returns cached data for Leeds:
def cache_key(url, params=None, headers=None):
parts = [url, json.dumps(sorted((params or {}).items()), default=str)]
# only headers that change the *content*, not Authorization or User-Agent
relevant = {k: v for k, v in (headers or {}).items() if k.lower() in {"accept", "accept-language"}}
parts.append(json.dumps(sorted(relevant.items())))
return hashlib.sha256("|".join(parts).encode()).hexdigest()[:16]Two rules. Include everything that varies the response body. Exclude credentials — a key containing a token means a rotated token invalidates a cache that is perfectly good, and it puts secrets in filenames.
2. Choose an expiry that matches the source
There is no universal TTL. The right value comes from how fast the source changes:
MAX_AGE = {
"boundaries": timedelta(days=365), # annual releases
"statistics": timedelta(days=90), # quarterly
"osm": timedelta(days=7), # continuous editing
"imagery": timedelta(days=30), # new scenes every few days
"geocode": timedelta(days=730), # addresses barely move
}Setting an expiry you have thought about is the point, not the specific number. A cache with no expiry is a stale-data generator; a cache with a one-hour expiry is not a cache.
3. Store the payload and the metadata separately
Two files per entry — <key>.bin and <key>.json — beats one combined file for a practical reason: you can inspect, count and audit the metadata without touching gigabytes of payload.
import pandas as pd
rows = [json.loads(p.read_text()) for p in CACHE.glob("*.json")]
inventory = pd.DataFrame(rows)[["retrieved", "bytes", "url"]]
inventory["mb"] = (inventory.pop("bytes") / 1e6).round(1)
print(inventory.sort_values("retrieved").to_string(index=False)) retrieved url mb
2026-08-20T09:14:02+00:00 https://service.pdok.nl/…gemeenten 2.4
2026-08-26T10:41:55+00:00 https://www.geoboundaries.org/…NLD 18.7
That table answers "what is my analysis actually built on" in one line.
4. Never cache a failure
This is the rule most caches get wrong. An empty result, a truncated response or an error page cached for a week is worse than no cache at all, because every rerun serves the bad answer instantly with no network call to correct it.
def validate_geojson(content):
payload = json.loads(content)
if "remark" in payload:
raise ValueError(f"partial result: {payload['remark']}")
matched = payload.get("numberMatched")
returned = payload.get("numberReturned", len(payload.get("features", [])))
if matched is not None and returned < matched:
raise ValueError(f"truncated: {returned} of {matched}")
if not payload.get("features"):
raise ValueError("no features")
return payloadValidate before writing to the cache. The checks are the ones from GeoJSON downloaded from an API is empty or truncated, and the cache is exactly where they pay off most.
5. Make the cache inspectable and clearable
def cache_stats(cache=CACHE):
entries = list(cache.glob("*.json"))
total = sum(p.with_suffix(".bin").stat().st_size for p in entries
if p.with_suffix(".bin").exists())
ages = [datetime.now(timezone.utc) - datetime.fromisoformat(json.loads(p.read_text())["retrieved"])
for p in entries]
print(f"{len(entries)} entries, {total / 1e6:.1f} MB, "
f"oldest {max(ages).days}d, newest {min(ages).days}d")7 entries, 41.3 MB, oldest 189d, newest 0d
A cache you cannot inspect is a cache you will eventually distrust and delete wholesale.
Code examples
Example 1 — a cache that validates, expires and records provenance
import hashlib
import json
from datetime import datetime, timedelta, timezone
from pathlib import Path
import requests
class Cache:
"""A small on-disk HTTP cache with an expiry, validation and an inventory."""
def __init__(self, root="data/cache", default_max_age=timedelta(days=7)):
self.root = Path(root)
self.root.mkdir(parents=True, exist_ok=True)
self.default_max_age = default_max_age
def _key(self, url, params):
blob = f"{url}|{json.dumps(sorted((params or {}).items()), default=str)}"
return hashlib.sha256(blob.encode()).hexdigest()[:16]
def get(self, url, params=None, *, max_age=None, validate=None,
label=None, licence=None, headers=None):
max_age = max_age or self.default_max_age
key = self._key(url, params)
body, meta = self.root / f"{key}.bin", self.root / f"{key}.json"
if body.exists() and meta.exists():
info = json.loads(meta.read_text())
age = datetime.now(timezone.utc) - datetime.fromisoformat(info["retrieved"])
if age < max_age:
print(f" hit {label or url[:44]:46} ({age.days}d)")
return body.read_bytes()
print(f" stale {label or url[:44]:46} ({age.days}d > {max_age.days}d)")
r = requests.get(url, params=params, headers=headers, timeout=300)
r.raise_for_status()
if validate is not None:
validate(r.content) # raises rather than caching a bad response
body.write_bytes(r.content)
meta.write_text(json.dumps({
"label": label,
"url": r.url,
"licence": licence,
"retrieved": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"bytes": len(r.content),
"sha256": hashlib.sha256(r.content).hexdigest(),
}, indent=2))
print(f" fetch {label or url[:44]:46} ({len(r.content) / 1e6:.1f} MB)")
return body.read_bytes()
def clear(self, older_than=None):
removed = 0
for meta in self.root.glob("*.json"):
info = json.loads(meta.read_text())
age = datetime.now(timezone.utc) - datetime.fromisoformat(info["retrieved"])
if older_than is None or age > older_than:
meta.with_suffix(".bin").unlink(missing_ok=True)
meta.unlink()
removed += 1
print(f"removed {removed} entries")
cache = Cache()
raw = cache.get(
"https://demo.pygeoapi.io/master/collections/lakes/items",
{"f": "json", "limit": 50},
max_age=timedelta(days=30),
validate=validate_geojson,
label="pygeoapi lakes",
licence="public domain",
)
raw = cache.get(
"https://demo.pygeoapi.io/master/collections/lakes/items",
{"f": "json", "limit": 50},
label="pygeoapi lakes",
) fetch pygeoapi lakes (0.1 MB)
hit pygeoapi lakes (0d)
Two identical calls, one network request. The validate hook is what stops a truncated response from being written, and the licence field is what makes the cache double as the provenance record from GIS data sources explained.
Example 2 — caching parsed objects, not just bytes
Re-parsing a 200 MB GeoJSON on every run is often slower than the download it replaced. Cache the parsed form in a fast format:
import geopandas as gpd
def cached_layer(cache, url, params=None, *, label, max_age=timedelta(days=30), **kw):
"""Return a GeoDataFrame, caching the parsed GeoPackage rather than the raw bytes."""
key = cache._key(url, params)
parsed = cache.root / f"{key}.gpkg"
if parsed.exists():
meta = json.loads((cache.root / f"{key}.json").read_text())
age = datetime.now(timezone.utc) - datetime.fromisoformat(meta["retrieved"])
if age < max_age:
print(f" parsed hit {label} ({age.days}d)")
return gpd.read_file(parsed)
raw = cache.get(url, params, max_age=max_age, label=label, **kw)
gdf = gpd.read_file(raw)
gdf.to_file(parsed, driver="GPKG")
return gdf
import time
for attempt in (1, 2):
start = time.perf_counter()
lakes = cached_layer(cache, "https://demo.pygeoapi.io/master/collections/lakes/items",
{"f": "json", "limit": 50}, label="lakes",
validate=validate_geojson)
print(f" attempt {attempt}: {len(lakes)} features in {time.perf_counter() - start:.2f}s") hit lakes (0d)
attempt 1: 25 features in 0.09s
parsed hit lakes (0d)
attempt 2: 25 features in 0.01s
GeoPackage reads are near-instant because the geometry is already binary and indexed. For large layers this second level of caching saves more time than the first.
Example 3 — an offline mode for reproducibility and CI
import os
class OfflineCache(Cache):
"""Raise instead of hitting the network — for CI, and for reproducing old results."""
def get(self, url, params=None, **kw):
key = self._key(url, params)
body = self.root / f"{key}.bin"
if not body.exists():
raise RuntimeError(
f"offline mode: no cache entry for {kw.get('label') or url}\n"
f" run once online to populate {body}"
)
print(f" offline {kw.get('label') or url[:44]}")
return body.read_bytes()
cache = OfflineCache() if os.environ.get("GIS_OFFLINE") else Cache()$ GIS_OFFLINE=1 python analysis.py
offline pygeoapi lakes
offline NLD ADM2 boundaries
This turns the cache into a correctness tool rather than only a speed one. In CI it means a test suite cannot silently depend on a live third-party service — an accidental network call becomes a failure rather than a flaky test. It is the same idea as making tests not read live files.
Explanation
Why "does the file exist" is not caching
if not path.exists(): download() has no notion of time, so the first download is permanent. It also has no notion of which request produced the file, so changing a parameter silently reuses the old answer.
Both failures are invisible. The script runs, the file loads, the analysis completes — against data that answers a different question, or answers this one as of eighteen months ago.
The three additions that fix it are small: hash the request into the filename, store the retrieval time, compare it against a deliberate maximum.
Why an expiry beats a conditional request
HTTP has ETag and If-Modified-Since, and where a server supports them they are excellent: a 304 Not Modified costs almost nothing and confirms your copy is current.
In practice most spatial services do not implement them usefully. Dynamically generated GeoJSON usually has no stable ETag, and Last-Modified often reflects the request rather than the data.
So an explicit TTL is the pragmatic default, with conditional requests as an optimisation where they work:
headers = {}
if info.get("etag"):
headers["If-None-Match"] = info["etag"]
r = requests.get(url, params=params, headers=headers, timeout=300)
if r.status_code == 304:
return body.read_bytes() # server confirms our copy is currentWhy caching failures is the worst outcome
A cache turns one request into many replays. That is the entire benefit, and it is also the entire risk: if the thing replayed is wrong, the cache multiplies the error and removes the mechanism that would have corrected it.
An empty GeoJSON cached under a seven-day TTL means a week of runs that produce no features and no error. The validate hook exists for exactly this, and it should be strict — refuse the response and let the caller fail loudly, rather than storing something questionable.
Why the cache is part of your provenance
Once the cache stores the URL, the retrieval time, the licence and a hash of the content, it is the provenance record. Anyone can ask what the analysis was built on and get an exact answer.
The hash is the underrated field. Refetch in three months, compare hashes, and you know instantly whether the source changed — without diffing geometry or trusting a "last updated" label on a portal page.
Edge cases or notes
- Do not put credentials in the cache key. A rotated token would invalidate perfectly good entries, and secrets end up in filenames.
- Truncate the hash for readability, not for security. Sixteen hex characters is ample for a cache namespace.
- Caches need a size limit if the process is long-lived. Evict the oldest entries, or clear anything older than the longest TTL you use.
- Two processes writing the same key can interleave. Write to a temporary file and
rename()— that is atomic on POSIX — if concurrency is possible. - The OSMnx cache has no expiry. It is keyed correctly but never invalidates, so treat it as a build artefact and delete it deliberately. See OSMnx download fails, hangs or times out.
- Cache geocoding results almost forever. Addresses do not move, and at one request per second the cache is what makes a large file feasible at all.
- Do not cache derived analysis under the same key as its inputs. Key derived outputs on a hash of the input hashes, so changing an input invalidates everything downstream.
- Commit the cache metadata, not the payload. The JSON sidecars are small, diffable and make a repository's data dependencies reviewable.
Internal links
- GIS data sources explained — provenance, which the cache metadata provides
- GeoJSON downloaded from an API is empty or truncated — the validations to run before caching
- How to download data from a WFS service in Python — a paged download worth caching
- How to download OpenStreetMap data in Python with OSMnx — a built-in cache with no expiry
- How to geocode addresses in Python — where caching is a hard requirement, not an optimisation
- How to test GIS code that reads files — offline mode in CI
- How to process only the files that changed — the same idea applied to a pipeline
- How to make a GIS script idempotent — reruns that cost nothing and change nothing
FAQ
Why is if not path.exists() not good enough?
It has no expiry and no request key. The first download becomes permanent, and changing a parameter silently reuses the old answer.
What should the cache key include?
Everything that changes the response body — the URL and all parameters, plus content-negotiation headers. Never credentials.
How long should entries live?
Match the source. Annual boundary releases can live a year; OpenStreetMap a week; geocoding results essentially forever. The important part is choosing deliberately.
Should I cache errors?
Never. A cached empty or truncated response is replayed instantly for the whole TTL with no network call to correct it. Validate before writing.
Should I cache the bytes or the parsed object?
Both, in two layers. The bytes save the download; a parsed GeoPackage saves the parse, which for large layers is the bigger cost.
How do I make my analysis reproducible?
Add an offline mode that raises instead of fetching. Then a run either uses exactly the cached inputs or fails, and CI cannot silently depend on a live service.
Should the cache go in version control?
The metadata sidecars, yes — they are small and make data dependencies reviewable. The payloads, no.