HTTP Caching for Spatial Data: ETags, Max-Age and Invalidation
On this page
Problem statement
Caching is the cheapest performance work available on any HTTP service, and spatial services benefit more than most: tiles are immutable for their content, feature responses repeat, and the payloads are large.
The measured difference on one real endpoint — 50 Natural Earth provinces as GeoJSON, ten clients at once (Example 3):
throughput p50 latency bytes on the wire
generate and send 21.9 rps 453.3 ms 1,010,943
answer 304 Not Modified 934.1 rps 8.7 ms 0
About forty times the throughput and a fiftieth of the latency, for an ETag header and a string comparison.
The reason it is so often missing is that caching looks like an infrastructure concern. It is not: Cache-Control, ETag and Vary are three response headers, and getting them right is a design decision about how stale each resource may be.
Quick answer
Three headers, chosen per resource:
from fastapi import Request, Response
import hashlib
def cached_response(body: bytes, request: Request, media_type: str,
max_age: int = 300, immutable: bool = False):
etag = '"' + hashlib.sha256(body).hexdigest()[:16] + '"'
cache_control = (f"public, max-age={max_age}, immutable" if immutable
else f"public, max-age={max_age}")
headers = {"ETag": etag, "Cache-Control": cache_control, "Vary": "Accept, Accept-Encoding"}
if request.headers.get("if-none-match") == etag:
return Response(status_code=304, headers=headers) # a 304 repeats the headers
return Response(body, media_type=media_type, headers=headers)The policy, chosen per resource:
tiles at a versioned URL public, max-age=31536000, immutable
tiles at a live URL public, max-age=60
feature queries public, max-age=300 + ETag
a downloadable file public, max-age=86400 + ETag
anything user-specific private, no-store
Staleness first, then ETags, max-age and versioned URLs
1. Decide how stale each resource may be
This is the only genuinely hard part, and it is a product question rather than a technical one. "How often does the data change?" is the wrong question; "how out of date may a response be before somebody is misled?" is the right one.
- Immutable — a tile at a versioned URL, a file with a content hash in its name. Never revalidate.
- Minutes — feature queries over slowly changing data.
- Seconds — live positions, editing sessions.
- Never cache — anything user-specific or authenticated, unless marked
private.
2. Use ETag for revalidation
An ETag is an opaque identifier for the response body. The client sends it back as If-None-Match, and the server answers 304 Not Modified with no body if it still matches.
The saving is the body plus the transfer, and — if the response is precomputed — the generation too. Measured, that was 1.01 MB and about 445 ms per request.
Compute it from the body when the body is cheap to produce, or from a data version when it is not:
etag = f'"{layer_version}-{query_hash}"'3. Use Cache-Control for the freshness window
max-age says how long a response may be reused without asking. During that window the client does not contact the server at all — which is strictly better than a 304, because there is no round trip.
Cache-Control: public, max-age=300
public allows shared caches (a CDN, a proxy) to store it; private restricts it to the browser. Use private for anything that depends on who is asking.
4. Version URLs so tiles can be immutable
immutable tells the client never to revalidate, even on a reload. It is only safe when the URL's content genuinely cannot change — which you achieve by putting a version in the path:
/tiles/v7/{z}/{x}/{y}.mvt public, max-age=31536000, immutable
A data update becomes /tiles/v8/…, and nothing has to be purged: the old URLs simply stop being requested. This is the cleanest invalidation strategy available, and it is why it is worth designing the URL for it from the start.
5. Set Vary correctly, or caches will serve the wrong thing
A shared cache keys on the URL. If the response depends on a request header — the format negotiated through Accept, the compression through Accept-Encoding — the cache must be told, or it will serve a GeoJSON response to a client that asked for Parquet.
Vary: Accept, Accept-Encoding
Keep the list short. Every header in Vary multiplies the number of cache entries; Vary: * disables caching entirely.
6. Compress, and let the cache know
Gzip is not caching, and it compounds with it. Measured on the same 50-province response in How to set cache headers on a spatial API: 1,010,943 bytes became 337,521 on the wire, a factor of 3.00.
Include Accept-Encoding in Vary so a cache does not serve a compressed body to a client that cannot decompress it.
Caching policies and what they buy
The data used on this page
The feature endpoint serves Natural Earth's 1:10m admin-1 states and provinces (provinces.gpkg, 4,596 polygons, 121 attributes). The setup creates the FastAPI app, loads the layer and defines build_features, the expensive step Example 2 defers until the ETag has missed. Example 3 serves that app with uvicorn on the same machine as the client.
import os
import geopandas as gpd
from fastapi import FastAPI
if not os.path.exists("provinces.gpkg"):
gpd.read_file("https://naciscdn.org/naturalearth/10m/cultural/ne_10m_admin_1_states_provinces.zip") \
.to_file("provinces.gpkg")
LAYER = gpd.read_file("provinces.gpkg")
LAYER_VERSION = "ne-10m-admin1"
app = FastAPI()
def build_features(bbox, limit):
subset = LAYER
if bbox:
x0, y0, x1, y1 = (float(v) for v in bbox.split(","))
subset = subset.cx[x0:x1, y0:y1]
return subset.iloc[:limit].to_json().encode()
print(len(LAYER))4596
Example 1 — a caching policy per resource type
from dataclasses import dataclass
from fastapi import Request, Response
import hashlib
@dataclass
class CachePolicy:
max_age: int
public: bool = True
immutable: bool = False
etag: bool = True
vary: tuple = ("Accept", "Accept-Encoding")
def headers(self, etag_value=None):
directives = ["public" if self.public else "private",
f"max-age={self.max_age}"]
if self.immutable:
directives.append("immutable")
headers = {"Cache-Control": ", ".join(directives)}
if self.vary:
headers["Vary"] = ", ".join(self.vary)
if etag_value:
headers["ETag"] = etag_value
return headers
POLICIES = {
"tile_versioned": CachePolicy(max_age=31_536_000, immutable=True, etag=False),
"tile_live": CachePolicy(max_age=60),
"features": CachePolicy(max_age=300),
"file_download": CachePolicy(max_age=86_400),
"user_specific": CachePolicy(max_age=0, public=False, etag=False),
}
def respond(body: bytes, request: Request, media_type: str, policy_name: str):
policy = POLICIES[policy_name]
etag = ('"' + hashlib.sha256(body).hexdigest()[:16] + '"') if policy.etag else None
if etag and request.headers.get("if-none-match") == etag:
return Response(status_code=304, headers=policy.headers(etag))
return Response(body, media_type=media_type, headers=policy.headers(etag))Example 2 — an ETag from a data version rather than the body
import hashlib
import json
def query_etag(layer_version: str, params: dict) -> str:
"""Cheap: no need to generate the body to know whether it changed."""
key = json.dumps({"v": layer_version, **params}, sort_keys=True)
return '"' + hashlib.sha256(key.encode()).hexdigest()[:16] + '"'
@app.get("/features")
def features(request: Request, bbox: str | None = None, limit: int = 50):
etag = query_etag(LAYER_VERSION, {"bbox": bbox, "limit": limit})
headers = {"ETag": etag, "Cache-Control": "public, max-age=300"}
if request.headers.get("if-none-match") == etag:
return Response(status_code=304, headers=headers)
body = build_features(bbox, limit) # only now do the work
return Response(body, media_type="application/geo+json", headers=headers)This is the version worth having on an expensive endpoint: the 304 path never touches the data at all. It requires a LAYER_VERSION that changes whenever the data does — a timestamp, a load id, a content hash computed at ingest.
Example 3 — measuring what caching buys on your own service
import asyncio
import statistics
import time
import httpx
async def measure(url, n=200, concurrency=10, headers=None):
limits = httpx.Limits(max_connections=concurrency)
async with httpx.AsyncClient(limits=limits, timeout=60) as client:
await client.get(url, headers=headers or {}) # warm
latencies = []
semaphore = asyncio.Semaphore(concurrency)
async def one():
async with semaphore:
started = time.perf_counter()
response = await client.get(url, headers=headers or {})
latencies.append(time.perf_counter() - started)
return response
started = time.perf_counter()
responses = await asyncio.gather(*[one() for _ in range(n)])
wall = time.perf_counter() - started
latencies.sort()
return {"status": responses[0].status_code,
"rps": round(n / wall, 1),
"p50_ms": round(statistics.median(latencies) * 1000, 1),
"bytes": int(responses[0].headers.get("content-length", 0))}
async def compare(url):
async with httpx.AsyncClient() as client:
etag = (await client.get(url)).headers.get("etag")
cold = await measure(url)
warm = await measure(url, headers={"If-None-Match": etag})
print(f"200 path {cold['rps']:7.1f} rps p50 {cold['p50_ms']:6.1f} ms "
f"{cold['bytes']:,} bytes")
print(f"304 path {warm['rps']:7.1f} rps p50 {warm['p50_ms']:6.1f} ms "
f"{warm['bytes']:,} bytes")
# here: Example 2's app, served on a local port
import os, threading, uvicorn
server = uvicorn.Server(uvicorn.Config(app, port=8768, log_level="warning"))
threading.Thread(target=server.run, daemon=True).start()
while not server.started:
time.sleep(0.1)
print(f"load average {os.getloadavg()[0]:.1f}")
asyncio.run(compare("http://127.0.0.1:8768/features?limit=50"))
server.should_exit = Trueload average 0.7
200 path 21.9 rps p50 453.3 ms 1,010,943 bytes
304 path 934.1 rps p50 8.7 ms 0 bytes
Saving the body versus saving the request
Why max-age beats ETag when both apply
An ETag saves the body; max-age saves the request. Within the freshness window the client does not contact the server at all, so the saving includes the round trip — which on a mobile network is frequently the dominant cost.
Use both: max-age for the window in which staleness is acceptable, and an ETag so that revalidation after the window is cheap. The measured 304 path at 934 requests per second is the revalidation case; the cached case costs the server nothing whatsoever.
Why immutable content is the strongest form
immutable removes revalidation even on a reload, which is otherwise the one action that bypasses max-age. It is only correct when the URL's content cannot change — and versioning the URL is how you make that true.
The consequence for tiles is significant: with a versioned prefix, a CDN never asks the origin about a tile again, and a data update is a deployment of a new prefix rather than a purge of millions of objects.
Why Vary is the header that causes silent bugs
A cache stores a response against a URL. If two clients ask for the same URL and receive different bodies — one GeoJSON, one Parquet, because they sent different Accept headers — the cache will serve whichever it stored to both.
Vary: Accept tells it to key on that header too. Forgetting it produces a bug that only appears through a cache, only for some clients, and is close to impossible to reproduce locally.
Why caching is a design decision rather than a deployment one
The freshness window encodes a product decision: how out of date a response may be. Nobody but the people who understand the data can answer that, and it is different per resource — tiles for a boundary layer can be immutable for a year, positions from a live feed for five seconds.
Putting a CDN in front of a service that sends no cache headers achieves very little, because the CDN's defaults are conservative. The headers are where the decision lives, and they belong next to the endpoint that knows what the data is.
no-store, private, weak ETags and reloads
no-storemeans never write it down;no-cachemeans revalidate before use. They are not synonyms.privatefor anything user-specific, or a shared cache may serve one user's data to another.- A weak ETag (
W/"…") allows semantically equivalent bodies to match; strong ETags require byte equality. Vary: *disables caching. Keep the list short and specific.- A reload bypasses
max-ageunless the response isimmutable. - Compress and cache — they compound; measured, gzip alone was 3.00×.
- 304 responses must repeat the ETag and any headers that would have varied.
- Version the URL rather than purging a CDN. Purging millions of tiles is slow and often incomplete.
Internal links
- How to set cache headers on a spatial API and tile service — the implementation
- Serving spatial data explained: files, features and tiles — where caching fits
- Dynamic tile server or static tiles: what to serve — versioned tile URLs
- Fixing a spatial API that is slow under load — caching as the first fix
- How to serve vector tiles from PostGIS with ST_AsMVT — caching dynamic tiles
- GeoJSON, vector tiles or Parquet: choosing an API response format — why
Vary: Acceptmatters - How to cache downloaded GIS data in Python — the client side
- Cloud-native geospatial explained — range requests and caching
FAQ
How much does HTTP caching actually save?
Measured on one endpoint: 934 requests per second and 8.7 ms for a 304 response, against 22 requests per second and 453 ms for generating and sending a 1 MB body.
What is the difference between ETag and max-age?
max-age says how long a response may be reused without asking, so it saves the whole round trip. ETag makes the ask cheap when it happens, by allowing a 304 with no body.
When can I use immutable?
When the URL's content cannot change — which you guarantee by putting a version in the path. Then a data update is a new prefix and nothing needs purging.
Why do I need Vary?
Because caches key on the URL. If the response depends on Accept or Accept-Encoding, a cache without Vary will serve the wrong format or a compressed body to a client that cannot read it.
What cache lifetime should feature queries use?
However stale a response may be before somebody is misled — usually minutes. It is a product decision, not a technical one, and it differs per resource.
Should I still compress if I cache?
Yes. They compound: caching removes requests, compression shrinks the ones that remain. Measured, gzip took a response from 1.01 MB to 338 kB.