Fixing 429 Too Many Requests When Geocoding

Problem statement

The batch job was running fine, and then:

requests.exceptions.HTTPError: 429 Client Error: Too Many Requests

Or worse, and more permanently:

403 Client Error: Forbidden

The two are different diagnoses with different fixes. A 429 says "you are going too fast right now" and is recoverable by slowing down. A 403 from a service like the public Nominatim instance usually says "you have breached the usage policy" and is not recoverable by retrying — it is an IP-level block that can outlast your patience and affects everyone sharing your address.

The instinct after either is to add threads or retries. Both make it worse.

Quick answer

Rate limit inside the client so the caller cannot exceed it, and back off exponentially on 429:

import time
import requests


class ThrottledClient:
    def __init__(self, base, user_agent, min_interval=1.1, max_attempts=5):
        self.base, self.min_interval, self.max_attempts = base, min_interval, max_attempts
        self._last = 0.0
        self.session = requests.Session()
        self.session.headers["User-Agent"] = user_agent      # never the default

    def get(self, path, **params):
        for attempt in range(self.max_attempts):
            gap = time.monotonic() - self._last
            if gap < self.min_interval:
                time.sleep(self.min_interval - gap)
            self._last = time.monotonic()

            r = self.session.get(f"{self.base}{path}", params=params, timeout=30)

            if r.status_code == 429:
                wait = self._retry_after(r) or min(60, 2 ** attempt * 5)
                print(f"429; sleeping {wait}s (attempt {attempt + 1})")
                time.sleep(wait)
                continue
            if r.status_code == 403:
                raise PermissionError(
                    "403 — this is a policy block, not a rate limit. Stop the job.")
            r.raise_for_status()
            return r.json()
        raise RuntimeError("still rate limited after backoff")

    @staticmethod
    def _retry_after(response):
        value = response.headers.get("Retry-After")
        if not value:
            return None
        return int(value) if value.isdigit() else 60

Three details do the work: the interval is enforced by the object, Retry-After is honoured when the server sends it, and a 403 stops the job instead of being retried.

429 against 403, limits in the client, retries

1. Tell 429 and 403 apart, and treat them differently

429 Too Many Requests is a rate limit. The service is asking you to slow down and will serve you again. Honour Retry-After if present, otherwise back off exponentially.

403 Forbidden, from a shared open service, is normally a policy block: wrong or missing User-Agent, parallel connections, or sustained bulk traffic. Retrying extends it. The correct response is to stop, fix the client, and — if the volume justified the traffic — move to your own instance or a paid provider.

Some commercial APIs use 403 for an invalid key and 429 for the quota; read the body, which usually says which.

2. Put the limit in the client, not in the loop

A time.sleep(1) inside a for loop is a rate limit that lasts exactly as long as nobody writes a second loop. A client that tracks its own last-request time cannot be made to breach the limit by a caller who forgot.

It also makes the cost visible: len(todo) * client.min_interval is the honest estimate of how long the job will take, printed before it starts.

3. Do not add threads

The arithmetic is tempting and wrong: 40,000 addresses at one per second is eleven hours; sixteen threads promise forty minutes. The policy limits total requests per second, so the parallel version is a violation that ends in a 403.

The legitimate ways to make it faster, in order:

1. Deduplicate on the normalised address. A 40,000-row file typically contains far fewer distinct addresses.
2. Cache, so a rerun costs nothing and only new rows hit the service.
3. Buy capacity — a commercial bulk endpoint, priced for the volume.
4. Self-host, where concurrency becomes a tuning parameter rather than a breach.

4. Make the job restartable so a block is not a disaster

If every result is written to a durable cache as it arrives, a 429 storm or a hard stop costs you the current request and nothing else. The next run recomputes the to-do list as distinct keys minus cached keys and resumes.

This is what makes stopping on a 403 an acceptable strategy rather than a catastrophe: the work already done is on disk.

5. Identify yourself properly

For services without an API key, the User-Agent is the identification. The default python-requests/2.x is blocked outright by several public services. Send an application name and a contact address:

User-Agent: acme-analytics/1.0 (gis@acme.example)

The value of a contact address is that the alternative to a silent block becomes an email.

6. Add jitter when several jobs share a limit

If two scheduled jobs hit the same API, both backing off by exactly 5, 10, 20 seconds will collide again on every retry. Add a random fraction:

wait = min(60, 2 ** attempt * 5) * (0.5 + random.random())

A token bucket for a geocoder

The service used on this page

These examples do not test their behaviour against the public Nominatim instance — deliberately breaching its limit to see what happens is exactly what its usage policy forbids. Instead a local server enforces the same published rule, one request a second: a faster request gets a 429 with Retry-After: 2, and a client that collects ten 429s in a row is blocked with a 403 for the rest of the run.

import http.server
import json
import socket
import threading
import time
from urllib.parse import parse_qs, urlparse


class PolicyServer(http.server.BaseHTTPRequestHandler):
    """One request a second per client; 429 when faster; 403 after ten 429s in a row."""
    state = {"last": 0.0, "strikes": 0, "blocked": False, "lock": threading.Lock(),
             "served": 0, "limited": 0}

    def do_GET(self):
        s = self.state
        with s["lock"]:
            now = time.monotonic()
            if s["blocked"]:
                code = 403
            elif now - s["last"] < 1.0:
                s["strikes"] += 1
                s["limited"] += 1
                s["blocked"] = s["strikes"] >= 10
                code = 403 if s["blocked"] else 429
            else:
                s["last"], s["strikes"] = now, 0
                s["served"] += 1
                code = 200
        self.send_response(code)
        if code == 429:
            self.send_header("Retry-After", "2")
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        query = parse_qs(urlparse(self.path).query).get("q", [""])[0]
        if code == 200:
            self.wfile.write(json.dumps([{"lat": "52.05", "lon": "-2.71", "query": query}]).encode())

    def log_message(self, *args):
        pass


def start_server():
    PolicyServer.state.update(last=0.0, strikes=0, blocked=False, served=0, limited=0)
    with socket.socket() as probe:
        probe.bind(("127.0.0.1", 0))
        port = probe.getsockname()[1]
    server = http.server.ThreadingHTTPServer(("127.0.0.1", port), PolicyServer)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return server, f"http://127.0.0.1:{port}"


server, BASE = start_server()
print("policy server on", BASE)
server.shutdown()

policy server on http://127.0.0.1:36403

What each approach gets from the same service

Five clients, twenty lookups each, against a fresh server: a plain loop with no pause, four threads each with its own correctly throttled client, the throttled client on its own, and the throttled client after the server has already been provoked into a 403.

import contextlib
import io
from concurrent.futures import ThreadPoolExecutor

import requests


def plain_loop(base, n=20):
    codes = [requests.get(f"{base}/search", params={"q": f"place {i}"}, timeout=10).status_code
             for i in range(n)]
    return {code: codes.count(code) for code in sorted(set(codes))}


def four_threads(base, n=20):
    clients = [ThrottledClient(base, "spatialworkflow.io guide", max_attempts=1) for _ in range(4)]
    def one(i):
        try:
            clients[i % 4].get("/search", q=f"place {i}")
            return 200
        except PermissionError:
            return 403
        except RuntimeError:
            return 429
    notices = io.StringIO()
    with contextlib.redirect_stdout(notices), ThreadPoolExecutor(4) as pool:
        codes = list(pool.map(one, range(n)))
    outcome = {code: codes.count(code) for code in sorted(set(codes))}
    outcome["backoff sleeps"] = notices.getvalue().count("sleeping")
    return outcome


def throttled(base, n=20):
    client, codes = ThrottledClient(base, "spatialworkflow.io guide"), []
    started = time.monotonic()
    for i in range(n):
        client.get("/search", q=f"place {i}")
        codes.append(200)
    return {200: len(codes), "rate": round(n / (time.monotonic() - started), 2)}


for name, run in (("plain loop", plain_loop), ("four throttled threads", four_threads),
                  ("one throttled client", throttled)):
    server, BASE = start_server()
    started = time.monotonic()
    outcome = run(BASE)
    print(f"{name:24} {outcome}  in {time.monotonic() - started:4.1f} s; "
          f"server served {PolicyServer.state['served']}, refused {PolicyServer.state['limited']}")
    server.shutdown()

plain loop               {200: 1, 403: 10, 429: 9}  in  0.0 s; server served 1, refused 10
four throttled threads   {200: 6, 429: 14, 'backoff sleeps': 14}  in  8.0 s; server served 6, refused 14
one throttled client     {200: 20, 'rate': 0.96}  in 20.9 s; server served 20, refused 0

And once blocked, retrying achieves nothing:

server, BASE = start_server()
plain_loop(BASE, n=15)                                   # provokes the block
try:
    ThrottledClient(BASE, "spatialworkflow.io guide").get("/search", q="Hereford")
except PermissionError as stop:
    print("after the block:", stop)
server.shutdown()

after the block: 403 — this is a policy block, not a rate limit. Stop the job.

Example 1 — a token bucket for a per-minute quota

import threading
import time


class TokenBucket:
    """For quotas expressed per minute or per day, where short bursts are allowed."""

    def __init__(self, rate_per_second, capacity=None):
        self.rate = rate_per_second
        self.capacity = capacity or max(1.0, rate_per_second)
        self.tokens = self.capacity
        self.updated = time.monotonic()
        self.lock = threading.Lock()

    def take(self, tokens=1.0):
        with self.lock:
            now = time.monotonic()
            self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
            self.updated = now
            if self.tokens < tokens:
                shortfall = (tokens - self.tokens) / self.rate
                time.sleep(shortfall)
                self.tokens = 0.0
                self.updated = time.monotonic()
            else:
                self.tokens -= tokens

bucket, stamps = TokenBucket(rate_per_second=0.5, capacity=5), []
started = time.monotonic()
for _ in range(10):
    bucket.take()
    stamps.append(round(time.monotonic() - started, 1))
print("request times, s:", stamps)

request times, s: [0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 4.0, 6.0, 8.0, 10.0]

A bucket suits "10,000 per day" better than a fixed interval, because it allows the burst the quota permits while still averaging out. Do not use one against a service whose policy is a hard per-second rate — there, the fixed interval is the policy.

Example 2 — a runner that survives being throttled

def run_with_limits(todo, client, cache, checkpoint_every=50):
    done = failed = 0
    for i, key in enumerate(todo, 1):
        try:
            cache.put(key, client.geocode(key))
            done += 1
        except PermissionError as exc:                 # 403: stop, cleanly
            print(f"\n{exc}\nstopping after {done:,} lookups; cache is intact")
            break
        except RuntimeError:                           # exhausted backoff
            failed += 1
            cache.put(key, None, status="error")
            if failed > 20:
                print("\ntoo many rate-limit failures — the service is not "
                      "willing to serve this job right now. Stopping.")
                break
        if i % checkpoint_every == 0:
            print(f"  {i:,}/{len(todo):,}  ok={done:,} failed={failed:,}")
    return done, failed

Run against the policy server with a client that asks too fast — every other request collides — and a small in-memory cache:

class DictCache(dict):
    def put(self, key, value, status="ok"):
        self[key] = (value, status)


class TooFast(ThrottledClient):
    def geocode(self, key):
        return self.get("/search", q=key)


server, BASE = start_server()
eager = TooFast(BASE, "spatialworkflow.io guide", min_interval=0.5, max_attempts=2)
cache, notices = DictCache(), io.StringIO()
started = time.monotonic()
with contextlib.redirect_stdout(notices):              # the client prints each backoff
    done, failed = run_with_limits([f"address {i}" for i in range(40)], eager, cache,
                                   checkpoint_every=10)
print(f"done {done}, failed {failed}, backoff sleeps {notices.getvalue().count('sleeping')}, "
      f"{time.monotonic() - started:.0f} s")
print("server:", {k: PolicyServer.state[k] for k in ("served", "limited", "blocked")})
server.shutdown()

done 40, failed 0, backoff sleeps 39, 98 s
server: {'served': 40, 'limited': 39, 'blocked': False}

Asking twice as fast as allowed did not finish sooner. Nearly every lookup was refused once, waited out the two-second Retry-After and succeeded on the retry, so forty lookups took longer than the forty-odd seconds a client spacing its requests 1.1 s apart needs — and the server never blocked it only because every retry succeeded.

The failure ceiling matters. A job that grinds through ten thousand 429s is not making progress, it is being told no repeatedly.

Example 3 — measuring your actual request rate

import collections
import time


class RateMonitor:
    """What rate are we really achieving, including all the sleeping?"""

    def __init__(self, window=60):
        self.times = collections.deque()
        self.window = window

    def tick(self):
        now = time.monotonic()
        self.times.append(now)
        while self.times and now - self.times[0] > self.window:
            self.times.popleft()

    @property
    def rate(self):
        if len(self.times) < 2:
            return 0.0
        # n timestamps span n - 1 intervals
        return (len(self.times) - 1) / max(1e-9, self.times[-1] - self.times[0])

    def report(self, limit):
        print(f"observed {self.rate:.2f}/s against a limit of {limit:.2f}/s"
              + ("  <- OVER" if self.rate > limit else ""))

server, BASE = start_server()
monitor, client = RateMonitor(), ThrottledClient(BASE, "spatialworkflow.io guide", min_interval=1.0)
with contextlib.redirect_stdout(io.StringIO()):
    for i in range(15):
        client.get("/search", q=f"place {i}")
        monitor.tick()
monitor.report(limit=1.0)
n, span = len(monitor.times), monitor.times[-1] - monitor.times[0]
print(f"counting timestamps instead of intervals would say {n / span:.3f}/s")
print("server refused", PolicyServer.state["limited"])
server.shutdown()

observed 0.70/s against a limit of 1.00/s
counting timestamps instead of intervals would say 0.750/s
server refused 3

Two things show here. A client spacing its requests exactly 1.0 s apart — the limit itself — still had some refused: the server measures arrivals, and arrivals jitter by a few milliseconds either way, which is why the client's default is 1.1 s. And the first version of rate divided the number of timestamps by the time they span; fifteen requests span fourteen intervals, so it over-reported every rate, and a client running at exactly the limit read as over it.

Measure rather than assume. A client that sleeps 1.0 s between requests but takes 0.15 s per request is running at 0.87/s, comfortably inside a 1/s limit; one that sleeps 1.0 s after starting the request can exceed it.

Fixed intervals and Retry-After

Why a fixed interval is safer than a rate calculation

A limit of "one per second" is enforced on arrival times at the server, not on your average. A client that sends ten requests in one second and then sleeps nine seconds averages 1/s and will still be throttled.

Sleeping to a minimum gap between requests makes the instantaneous rate the same as the average, which is what the server measures. It costs a little throughput and removes the whole class of burst-related blocks.

Why Retry-After is worth honouring

When a server sends Retry-After, it is telling you exactly when it will serve you again. Ignoring it and using your own backoff means either waiting longer than necessary or — more often — retrying too early and being counted as another violation.

It arrives as either a number of seconds or an HTTP date. Handle both, and cap whatever you compute so a misconfigured header cannot park your job for a day.

Why a 403 needs a stop rather than a retry

Rate limits are per-request decisions; policy blocks are per-client state. Once a shared service has decided your client is misbehaving, further requests confirm the decision rather than testing it, and on some services extend the block.

Stopping is also the only response that lets you fix the cause. A job that keeps retrying obscures the reason it was blocked, and the fix — a proper User-Agent, a slower cadence, deduplication, or moving off the shared service — never gets made.

Why deduplication beats every other speed-up

Rate limiting bounds requests, not rows. A file with 40,000 rows and 12,000 distinct normalised addresses needs 12,000 requests, and if 9,000 are already cached from last month it needs 3,000 — under an hour at one per second.

That ordering is worth internalising: reduce the number of questions before trying to ask them faster. It is also the only speed-up that no usage policy objects to.

Response bodies, concurrency and shared IPs

• Check the response body on a 429. Some services distinguish a per-second limit from a monthly quota, and only one of them is fixable by waiting.
• Concurrency limits exist separately from rate limits. Four parallel workers can fail where sixteen sequential requests succeed.
• Shared IPs share limits. Office NAT, CI runners and cloud egress addresses are often already spending the quota.
• Retry-After can be an HTTP date, not just a number of seconds.
• Cap the backoff. Exponential growth with no ceiling parks a pipeline for hours.
• Add jitter when more than one job can hit the same limit.
• Log the status code, not just "failed". 429, 403, 502 and an empty result need different responses.
• A block can outlive the job. Test with a single request before re-launching a batch.

Internal links

• How to geocode with Nominatim from Python without being blocked — the policy and the compliant client
• How to batch geocode thousands of addresses in Python — deduplication and checkpointing
• How to cache geocoding results so a rerun costs nothing — the cache that makes retries unnecessary
• How to retry flaky steps in a GIS pipeline — the general backoff pattern
• My geocoder returns no match for addresses that exist — the failure that is not a rate limit
• How to build an offline geocoder from open address data — removing the limit entirely
• Fixing an API that returns empty or truncated GeoJSON — the same discipline for data APIs
• How to make a batch job resumable — surviving a mid-run stop

FAQ

What is the difference between 429 and 403?

429 is a rate limit — slow down and retry. 403 from a shared open service is usually a policy block on your IP, which retrying makes worse. Stop, fix the client, and consider self-hosting.

How fast can I geocode against the public Nominatim service?

About one request per second, sequentially, with a genuine User-Agent. Bulk geocoding is explicitly discouraged regardless of pacing.

Will threads make my batch faster?

No — they will get you blocked. The limit is on total requests per second. Deduplicate, cache, then buy capacity or self-host.

Should I honour the Retry-After header?

Yes. It tells you exactly when the server will serve you again. Cap whatever value you compute so a bad header cannot stall the pipeline for hours.

How do I recover from a block?

Stop the job, fix the cause — User-Agent, cadence, concurrency — and wait. Test with a single request before restarting a batch. The completed work should already be in the cache.

What rate limit should I set for my own instance?

Whatever your hardware sustains. Measure the achieved rate rather than assuming it; a client that sleeps one second per request but spends 0.15 s in flight is running at 0.87 requests per second.