How to Batch Geocode Thousands of Addresses in Python
On this page
Problem statement
Geocoding one address is a three-line script. Geocoding forty thousand is a different problem, and the differences are all operational:
- At one request per second — the public Nominatim rate — 40,000 rows take eleven hours, and any crash starts them again.
- The same address appears many times in a real customer file, so a large fraction of those requests are for answers you already have.
- Rate limits, transient 502s and a laptop lid closing are not exceptional events over eleven hours; they are certainties.
- Half the failures are silent: a coordinate comes back, at the wrong precision, and looks exactly like a success.
A batch geocoder is therefore a small pipeline: deduplicate, cache, throttle, retry, checkpoint, and report — with the actual API call as the least interesting part.
Quick answer
Geocode distinct addresses, not rows, and write every result to disk as it arrives:
import pandas as pd
def batch_geocode(df, address_col, client, cache):
"""Rows -> distinct keys -> cached lookups -> rows again."""
keys = df[address_col].map(normalise) # your normaliser
todo = sorted(set(keys) - cache.known())
print(f"{len(df):,} rows, {keys.nunique():,} distinct, {len(todo):,} to fetch")
for i, key in enumerate(todo, 1):
cache.put(key, client.geocode(key)) # client throttles itself
if i % 100 == 0:
print(f" {i:,}/{len(todo):,}")
results = keys.map(cache.get)
return df.assign(
lat=[r["lat"] if r else None for r in results],
lon=[r["lon"] if r else None for r in results],
precision=[r["precision"] if r else None for r in results],
)The first print is the one that saves money. Measured on Herefordshire's 1,783 registered food businesses, there were 1,443 distinct addresses, so a fifth of the lookups were never needed; the first pass took 38.5 minutes, and a restart with the cache in place fetched nothing at all (Example 2).
What you get — from The data used on this page
1,783 food businesses, 1,445 distinct address strings
Deduplicate on the normalised address, cache first, then run
1. Deduplicate on the normalised address, not the raw string
Normalisation is what makes the deduplication work: case, accents, punctuation and whitespace all vary without changing the place. How much it adds depends on who typed the file. On the council's own register it added almost nothing — 1,445 distinct raw strings became 1,443 normalised keys — because one system entered every address; a file merged from several sources gains far more.
Count both before you fetch anything: the difference is free information about how clean the file is.
2. Make the cache the primary data structure
Not a decorator on a function — a table on disk with a schema:
key the normalised query
response the provider's raw JSON, untouched
provider which service answered
fetched_at when
status ok | no_match | error
Storing the raw response means a change to your interpretation logic — a new precision mapping, a stricter filter — is a re-parse rather than a re-fetch. Storing misses means a permanently unmatchable address is asked about once, not once per run.
3. Throttle inside the client
The rate limit belongs to the client object, not to the loop. A client that enforces its own minimum interval cannot be made to breach the limit by a caller who forgot, and it makes the loop honest about the time the job will take.
Do not reach for threads. The public Nominatim policy limits total requests per second, so parallelism converts a slow compliant job into a fast blocked one. Legitimate speed comes from deduplication, caching, and — above a few hundred thousand lookups — running your own instance.
4. Checkpoint continuously, not at the end
Any long job must be restartable from where it stopped. Because the cache is written on every result, the restart logic is already there: recompute set(keys) - cache.known() and the finished work is skipped automatically.
That is the whole benefit of making the cache primary. There is no separate checkpoint file to keep in sync.
5. Classify every result, and report the classes
A batch job that reports "40,000 geocoded" has said nothing. The useful summary is by precision level and status:
postcode 697 39.1%
rooftop 639 35.8%
street 394 22.1%
region 42 2.4% <- unusable
locality 10 0.6%
no_match 1 0.1%
That is the county's register, and it carries a warning of its own. Checked against the coordinates the FSA publishes, the "rooftop" answers were a median 50 m out and one in ten over 627 m — worse than the postcode answers, at a median 44 m — and some "rooftop" and "street" matches were at the other end of Britain. A name that matches a building is not the building. (The FSA's own points are largely postcode-based, which flatters the postcode class; the far-away rooftops are errors regardless.)
Two of those lines are a work queue. Print them at the end of every run and the file's quality stops being a surprise.
6. Decide what happens to the unusable rows before you start
Drop, keep-and-label, or review. Write the choice into the config next to the analysis, because it differs per analysis: a national choropleth can live with locality centroids and a delivery route cannot.
The cache as a table, and the runner
The data used on this page
A real "customer file": every food business the Food Standards Agency lists for Herefordshire, with the address the council holds and the coordinates the FSA publishes for it — which gives each geocoded answer something to be checked against. The client below uses the public Nominatim service at its limit of one request a second, with a User-Agent that identifies the job. The register is a copy of the API's answer saved on 10 October 2026, so it stays the one measured here.
import json
import os
import re
import pandas as pd
import requests
if not os.path.exists("herefordshire.json"):
# live: requests.get("https://api.ratings.food.gov.uk/Establishments", headers={"x-api-version": "2"},
# params={"localAuthorityId": 383, "pageSize": 5000}, timeout=300) # Herefordshire
# read instead as the API answered on 10 October 2026, so the register stays put
reply = requests.get("https://data.spatialworkflow.io/fhrs-api-383-2026-10-10.json", timeout=300)
reply.raise_for_status()
json.dump(reply.json()["establishments"], open("herefordshire.json", "w"))
est = pd.DataFrame(json.load(open("herefordshire.json")))
lines = ["AddressLine1", "AddressLine2", "AddressLine3", "AddressLine4", "PostCode"]
est["address"] = est[lines].apply(lambda row: ", ".join(x for x in row if x), axis=1)
est["fsa_lat"] = est["geocode"].map(lambda g: float(g["latitude"]) if g and g.get("latitude") else None)
est["fsa_lon"] = est["geocode"].map(lambda g: float(g["longitude"]) if g and g.get("longitude") else None)
print(f"{len(est):,} food businesses, {est['address'].nunique():,} distinct address strings")1,783 food businesses, 1,445 distinct address strings
The pieces the runner below expects — a normaliser, a throttled client, and an interpretation of Nominatim's answer as a precision class:
import time
POSTCODE = re.compile(r"\b([a-z]{1,2}\d[a-z\d]?) ?(\d[a-z]{2})\b")
PRECISION_RANK = {"rooftop": 0, "street": 1, "postcode": 2, "locality": 3, "region": 4,
"no_match": 5}
def normalise(address):
text = re.sub(r"[^\w\s]", " ", address.lower())
return re.sub(r"\s+", " ", text).strip()
class NominatimClient:
min_interval = 1.1 # seconds between requests
def __init__(self, agent):
self.session = requests.Session()
self.session.headers["User-Agent"] = agent
self.last = 0.0
def _get(self, query):
wait = self.min_interval - (time.monotonic() - self.last)
if wait > 0:
time.sleep(wait)
self.last = time.monotonic()
reply = self.session.get("https://nominatim.openstreetmap.org/search", timeout=60,
params={"q": query, "format": "jsonv2", "limit": 1,
"countrycodes": "gb"})
if reply.status_code in (403, 429):
raise PermissionError(reply.status_code)
reply.raise_for_status()
return reply.json()
def search(self, key):
"""The whole address first; the postcode alone if that finds nothing."""
hits = self._get(key)
if hits:
return hits
postcode = POSTCODE.search(key)
return self._get(" ".join(postcode.groups())) if postcode else []
def interpret(hits):
hit = hits[0]
kind = hit.get("addresstype", "")
precision = ("rooftop" if kind in {"building", "amenity", "shop", "tourism", "leisure", "office",
"craft", "house", "place"}
else "street" if kind == "road"
else "postcode" if kind == "postcode"
else "locality" if kind in {"village", "hamlet", "town", "city", "suburb",
"neighbourhood", "isolated_dwelling", "farm", "locality"}
else "region")
return {"lat": float(hit["lat"]), "lon": float(hit["lon"]), "precision": precision,
"addresstype": kind}Example 1 — the cache, as a table
import json
import sqlite3
from datetime import datetime, timezone
class GeocodeCache:
def __init__(self, path="geocode_cache.sqlite", provider="nominatim"):
self.provider = provider
self.db = sqlite3.connect(path)
self.db.execute("""
create table if not exists geocode (
key text not null, provider text not null,
response text, status text not null, fetched_at text not null,
primary key (key, provider))""")
self.db.commit()
def known(self) -> set[str]:
rows = self.db.execute(
"select key from geocode where provider = ?", (self.provider,))
return {r[0] for r in rows}
def get(self, key):
row = self.db.execute(
"select response, status from geocode where key = ? and provider = ?",
(key, self.provider)).fetchone()
if not row or row[1] != "ok":
return None
return json.loads(row[0])
def put(self, key, result, status=None):
status = status or ("ok" if result else "no_match")
self.db.execute(
"insert or replace into geocode values (?, ?, ?, ?, ?)",
(key, self.provider, json.dumps(result) if result else None,
status, datetime.now(timezone.utc).isoformat(timespec="seconds")))
self.db.commit()
def stats(self):
return dict(self.db.execute(
"select status, count(*) from geocode where provider = ? group by 1",
(self.provider,)).fetchall())One commit() per row is deliberate. It is slower than batching and it means a hard kill loses at most one lookup — which, at a second per lookup, is the right trade.
Example 2 — the runner, with progress, retry and a final report
import time
from collections import Counter
def run_batch(df, address_col, client, cache, normalise, report_every=100):
keys = df[address_col].fillna("").map(normalise)
distinct = sorted(k for k in set(keys) if k)
todo = [k for k in distinct if k not in cache.known()]
print(f"rows {len(df):,}")
print(f"distinct {len(distinct):,} ({100 * len(distinct) / len(df):.1f}% of rows)")
print(f"already known {len(distinct) - len(todo):,}")
print(f"to fetch {len(todo):,} (~{len(todo) * client.min_interval / 3600:.1f} h)")
started, errors = time.monotonic(), 0
for i, key in enumerate(todo, 1):
try:
hits = client.search(key)
cache.put(key, interpret(hits) if hits else None)
except PermissionError:
print("blocked by the provider — stopping cleanly, cache is intact")
break
except Exception as exc: # transient: record and move on
errors += 1
cache.put(key, None, status="error")
print(f" ! {key[:50]}: {type(exc).__name__}")
if i % report_every == 0:
rate = i / (time.monotonic() - started)
print(f" {i:,}/{len(todo):,} {rate:.2f}/s eta "
f"{(len(todo) - i) / rate / 60:.0f} min")
resolved = keys.map(cache.get)
out = df.assign(
lat=[r["lat"] if r else None for r in resolved],
lon=[r["lon"] if r else None for r in resolved],
precision=[r["precision"] if r else "no_match" for r in resolved],
)
print("\nprecision breakdown")
for level, n in Counter(out["precision"]).most_common():
print(f" {level:10s} {n:7,} {100 * n / len(out):5.1f}%")
print(f"errors this run: {errors}")
return outThe whole county, run twice. The first run fetches every distinct address at the rate limit; the second is the restart after a crash, with the cache already full:
client = NominatimClient("spatialworkflow.io guide (batch geocoding example)")
cache = GeocodeCache("geocode_cache.sqlite")
for attempt in ("first run", "second run"):
print(f"--- {attempt}")
started = time.monotonic()
result = run_batch(est, "address", client, cache, normalise, report_every=500)
print(f"took {(time.monotonic() - started) / 60:.1f} min")--- first run
rows 1,783
distinct 1,443 (80.9% of rows)
already known 0
to fetch 1,443 (~0.4 h)
500/1,443 0.77/s eta 20 min
1,000/1,443 0.67/s eta 11 min
precision breakdown
postcode 697 39.1%
rooftop 639 35.8%
street 394 22.1%
region 42 2.4%
locality 10 0.6%
no_match 1 0.1%
errors this run: 0
took 38.5 min
--- second run
rows 1,783
distinct 1,443 (80.9% of rows)
already known 1,443
to fetch 0 (~0.0 h)
precision breakdown
postcode 697 39.1%
rooftop 639 35.8%
street 394 22.1%
region 42 2.4%
locality 10 0.6%
no_match 1 0.1%
errors this run: 0
took 0.0 min
And the part a batch report should always end with: how far the answers are from where the business actually is, by precision class.
import numpy as np
checked = result.dropna(subset=["lat", "fsa_lat"]).copy()
lat0 = np.radians(checked["fsa_lat"])
dy = (checked["lat"] - checked["fsa_lat"]) * 111_320
dx = (checked["lon"] - checked["fsa_lon"]) * 111_320 * np.cos(lat0)
checked["error_m"] = np.hypot(dx, dy)
summary = checked.groupby("precision")["error_m"].describe(percentiles=[0.5, 0.9])
print(summary[["count", "50%", "90%"]].round(0).sort_values("50%").to_string()) count 50% 90%
precision
postcode 610.0 44.0 257.0
rooftop 327.0 50.0 627.0
street 383.0 55.0 328.0
region 33.0 145.0 2839.0
locality 9.0 299.0 2259.0
Example 3 — turning the result into a GeoDataFrame, safely
import geopandas as gpd
def to_geodataframe(df, worst_allowed="street", crs="EPSG:4326"):
"""Only rows that met the precision floor become geometry."""
usable = df["precision"].map(PRECISION_RANK).fillna(9) <= PRECISION_RANK[worst_allowed]
dropped = (~usable).sum()
if dropped:
print(f"excluding {dropped:,} rows below '{worst_allowed}' precision")
good = df[usable & df["lat"].notna()].copy()
gdf = gpd.GeoDataFrame(
good,
geometry=gpd.points_from_xy(good["lon"], good["lat"]), # x, y — lon first
crs=crs,
)
gdf.attrs["excluded_rows"] = int(dropped)
return gdfgdf = to_geodataframe(result, worst_allowed="street")
print(f"{len(gdf):,} of {len(result):,} rows became points; bounds {gdf.total_bounds.round(2).tolist()}")excluding 750 rows below 'street' precision
1,033 of 1,783 rows became points; bounds [-4.3, 51.23, -1.12, 55.85]
points_from_xy(lon, lat) — in that order — is worth writing out every time. Swapping them produces points off the coast of Africa or, more insidiously, in a plausible-looking wrong country.
Why deduplication saves the most
Why deduplication is the largest single saving
Address files repeat because the world does: households order twice, businesses have many contacts at one site, and the same depot appears on every delivery row. A file where distinct addresses are 30% of rows is ordinary.
Deduplicating on the normalised key rather than the raw string compounds the effect, because the same address written three ways collapses to one lookup. Every subsequent optimisation — caching, self-hosting, buying a bulk tier — operates on a smaller number.
Why the cache has to store failures
An address that cannot be geocoded today cannot be geocoded tomorrow either, unless the reference data changed. Without negative caching, every run pays full price for the permanently unmatchable rows — which are, by definition, the slowest ones to fail.
Store the status and the timestamp, and expire negatives after a few months rather than never. Reference data does improve.
Why parallelism is the wrong first answer
The arithmetic that tempts everyone: 40,000 rows at 1/s is eleven hours; with sixteen threads it is forty minutes — and the county's 1,443 addresses, most needing two requests, took 38.5 minutes at the limit. The policy the public service enforces is on total requests per second, so the sixteen-thread version is a violation that ends in a 403 for your whole IP address.
The ordering that works: deduplicate (a fifth of the rows in the county's register), cache (a rerun costs nothing), and only then consider capacity — a paid bulk endpoint, or your own Nominatim instance where concurrency is a tuning parameter rather than a breach.
Why the precision breakdown belongs in the run output
A batch job's log is the only place most people will ever see the quality of the geocoding. If it prints a single count, the file's 5% locality centroids travel silently into every downstream analysis.
Printing the breakdown makes the trade-off visible at the moment it is made, and it turns "the geocoding is done" into "the geocoding is done, and 750 rows need a decision".
Empty addresses and structured queries
- Empty and near-empty addresses match countries and regions with high scores. Filter rows with fewer than about three tokens before sending.
- Country codes narrow the search and remove most cross-border mismatches: pass
countrycodeswhen you know it. - Structured queries beat free text if you have already parsed the components.
- A run that takes hours needs
nohupor a scheduler, not a laptop. - Commit per row. The cost is invisible next to a network round trip and it makes kills safe.
- Do not sort the input by anything meaningful — if the job stops early, a random order leaves you with a representative sample rather than only the As.
- Watch for stacked coordinates afterwards: many rows sharing one point is the signature of a fallback.
- Re-geocoding is a new measurement. Store the provider and date, or a rerun will silently change coordinates.
Internal links
- How to geocode with Nominatim from Python without being blocked — the throttled client this uses
- How to cache geocoding results so a rerun costs nothing — the cache in detail
- How to parse and normalise addresses in Python — the normaliser that makes deduplication work
- Rooftop, interpolated or centroid: geocoding precision levels — the breakdown the runner prints
- How to validate geocoding results before you trust them — what to do with the output
- Fixing 429 too many requests when geocoding — when throttling was not enough
- How to make a batch job resumable — the same checkpoint pattern for files
- How to build an offline geocoder from open address data — when the volume outgrows any API
FAQ
How long does it take to geocode 40,000 addresses?
Against the public Nominatim service at one request per second, about eleven hours before deduplication — longer if a fallback query is needed. Measured, 1,443 distinct addresses with a postcode fallback took 38.5 minutes; the re-run with a cache took none.
Can I use threads to speed it up?
Not against a service whose policy limits total requests per second; that is how you get a 403 for your whole IP. Deduplicate, cache, then self-host if the volume justifies it.
Should I cache addresses that failed to geocode?
Yes. They will fail again, and they are the slowest rows to fail. Store the status and expire negative entries after a few months.
How do I make the job resumable?
Write every result to the cache as it arrives, and compute the to-do list as distinct keys minus cached keys. There is then no separate checkpoint to maintain.
What should the batch job print at the end?
The breakdown by precision level and status. A single "geocoded 40,000" hides the rows that are unusable, and those are the reason the job existed.
Do I geocode rows or distinct addresses?
Distinct normalised addresses, then join the results back onto the rows. It is usually the single largest saving in the pipeline.