🔧 Fix Geocoding & addresses Pipelines & automation Geocoding & Addresses · step 17 of 20 Updated

Fixing 429 Too Many Requests When Geocoding

Problem statement

The batch job was running fine, and then:

requests.exceptions.HTTPError: 429 Client Error: Too Many Requests

Or worse, and more permanently:

403 Client Error: Forbidden

The two are different diagnoses with different fixes. A 429 says "you are going too fast right now" and is recoverable by slowing down. A 403 from a service like the public Nominatim instance usually says "you have breached the usage policy" and is not recoverable by retrying — it is an IP-level block that can outlast your patience and affects everyone sharing your address.

The instinct after either is to add threads or retries. Both make it worse.

Quick answer

Rate limit inside the client so the caller cannot exceed it, and back off exponentially on 429:

import time
import requests


class ThrottledClient:
    def __init__(self, base, user_agent, min_interval=1.1, max_attempts=5):
        self.base, self.min_interval, self.max_attempts = base, min_interval, max_attempts
        self._last = 0.0
        self.session = requests.Session()
        self.session.headers["User-Agent"] = user_agent      # never the default

    def get(self, path, **params):
        for attempt in range(self.max_attempts):
            gap = time.monotonic() - self._last
            if gap < self.min_interval:
                time.sleep(self.min_interval - gap)
            self._last = time.monotonic()

            r = self.session.get(f"{self.base}{path}", params=params, timeout=30)

            if r.status_code == 429:
                wait = self._retry_after(r) or min(60, 2 ** attempt * 5)
                print(f"429; sleeping {wait}s (attempt {attempt + 1})")
                time.sleep(wait)
                continue
            if r.status_code == 403:
                raise PermissionError(
                    "403 — this is a policy block, not a rate limit. Stop the job.")
            r.raise_for_status()
            return r.json()
        raise RuntimeError("still rate limited after backoff")

    @staticmethod
    def _retry_after(response):
        value = response.headers.get("Retry-After")
        if not value:
            return None
        return int(value) if value.isdigit() else 60

Three details do the work: the interval is enforced by the object, Retry-After is honoured when the server sends it, and a 403 stops the job instead of being retried.

Two panels contrasting HTTP 403 policy blocks with HTTP 429 rate limits.
Only one of these is a signal to slow down. The other is a signal to stop.

Step-by-step solution

1. Tell 429 and 403 apart, and treat them differently

429 Too Many Requests is a rate limit. The service is asking you to slow down and will serve you again. Honour Retry-After if present, otherwise back off exponentially.

403 Forbidden, from a shared open service, is normally a policy block: wrong or missing User-Agent, parallel connections, or sustained bulk traffic. Retrying extends it. The correct response is to stop, fix the client, and — if the volume justified the traffic — move to your own instance or a paid provider.

Some commercial APIs use 403 for an invalid key and 429 for the quota; read the body, which usually says which.

2. Put the limit in the client, not in the loop

A time.sleep(1) inside a for loop is a rate limit that lasts exactly as long as nobody writes a second loop. A client that tracks its own last-request time cannot be made to breach the limit by a caller who forgot.

It also makes the cost visible: len(todo) * client.min_interval is the honest estimate of how long the job will take, printed before it starts.

3. Do not add threads

The arithmetic is tempting and wrong: 40,000 addresses at one per second is eleven hours; sixteen threads promise forty minutes. The policy limits total requests per second, so the parallel version is a violation that ends in a 403.

The legitimate ways to make it faster, in order:

  1. Deduplicate on the normalised address. A 40,000-row file typically contains far fewer distinct addresses.
  2. Cache, so a rerun costs nothing and only new rows hit the service.
  3. Buy capacity — a commercial bulk endpoint, priced for the volume.
  4. Self-host, where concurrency becomes a tuning parameter rather than a breach.

4. Make the job restartable so a block is not a disaster

If every result is written to a durable cache as it arrives, a 429 storm or a hard stop costs you the current request and nothing else. The next run recomputes the to-do list as distinct keys minus cached keys and resumes.

This is what makes stopping on a 403 an acceptable strategy rather than a catastrophe: the work already done is on disk.

5. Identify yourself properly

For services without an API key, the User-Agent is the identification. The default python-requests/2.x is blocked outright by several public services. Send an application name and a contact address:

User-Agent: acme-analytics/1.0 ([email protected])

The value of a contact address is that the alternative to a silent block becomes an email.

6. Add jitter when several jobs share a limit

If two scheduled jobs hit the same API, both backing off by exactly 5, 10, 20 seconds will collide again on every retry. Add a random fraction:

wait = min(60, 2 ** attempt * 5) * (0.5 + random.random())
Five options for speeding up a geocoding batch, with threading marked as a violation.
The first two are free and no usage policy objects to either.

Code examples

Example 1 — a token bucket for a per-minute quota 

import threading
import time


class TokenBucket:
    """For quotas expressed per minute or per day, where short bursts are allowed."""

    def __init__(self, rate_per_second, capacity=None):
        self.rate = rate_per_second
        self.capacity = capacity or max(1.0, rate_per_second)
        self.tokens = self.capacity
        self.updated = time.monotonic()
        self.lock = threading.Lock()

    def take(self, tokens=1.0):
        with self.lock:
            now = time.monotonic()
            self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
            self.updated = now
            if self.tokens < tokens:
                shortfall = (tokens - self.tokens) / self.rate
                time.sleep(shortfall)
                self.tokens = 0.0
                self.updated = time.monotonic()
            else:
                self.tokens -= tokens

A bucket suits "10,000 per day" better than a fixed interval, because it allows the burst the quota permits while still averaging out. Do not use one against a service whose policy is a hard per-second rate — there, the fixed interval is the policy.

Example 2 — a runner that survives being throttled 

def run_with_limits(todo, client, cache, checkpoint_every=50):
    done = failed = 0
    for i, key in enumerate(todo, 1):
        try:
            cache.put(key, client.geocode(key))
            done += 1
        except PermissionError as exc:                 # 403: stop, cleanly
            print(f"\n{exc}\nstopping after {done:,} lookups; cache is intact")
            break
        except RuntimeError:                           # exhausted backoff
            failed += 1
            cache.put(key, None, status="error")
            if failed > 20:
                print("\ntoo many rate-limit failures — the service is not "
                      "willing to serve this job right now. Stopping.")
                break
        if i % checkpoint_every == 0:
            print(f"  {i:,}/{len(todo):,}  ok={done:,} failed={failed:,}")
    return done, failed

The failure ceiling matters. A job that grinds through ten thousand 429s is not making progress, it is being told no repeatedly.

Example 3 — measuring your actual request rate 

import collections
import time


class RateMonitor:
    """What rate are we really achieving, including all the sleeping?"""

    def __init__(self, window=60):
        self.times = collections.deque()
        self.window = window

    def tick(self):
        now = time.monotonic()
        self.times.append(now)
        while self.times and now - self.times[0] > self.window:
            self.times.popleft()

    @property
    def rate(self):
        if len(self.times) < 2:
            return 0.0
        return len(self.times) / max(1e-9, self.times[-1] - self.times[0])

    def report(self, limit):
        print(f"observed {self.rate:.2f}/s against a limit of {limit:.2f}/s"
              + ("  <- OVER" if self.rate > limit else ""))

Measure rather than assume. A client that sleeps 1.0 s between requests but takes 0.15 s per request is running at 0.87/s, comfortably inside a 1/s limit; one that sleeps 1.0 s after starting the request can exceed it.

Explanation

Why a fixed interval is safer than a rate calculation

A limit of "one per second" is enforced on arrival times at the server, not on your average. A client that sends ten requests in one second and then sleeps nine seconds averages 1/s and will still be throttled.

Sleeping to a minimum gap between requests makes the instantaneous rate the same as the average, which is what the server measures. It costs a little throughput and removes the whole class of burst-related blocks.

Why Retry-After is worth honouring

When a server sends Retry-After, it is telling you exactly when it will serve you again. Ignoring it and using your own backoff means either waiting longer than necessary or — more often — retrying too early and being counted as another violation.

It arrives as either a number of seconds or an HTTP date. Handle both, and cap whatever you compute so a misconfigured header cannot park your job for a day.

Why a 403 needs a stop rather than a retry

Rate limits are per-request decisions; policy blocks are per-client state. Once a shared service has decided your client is misbehaving, further requests confirm the decision rather than testing it, and on some services extend the block.

Stopping is also the only response that lets you fix the cause. A job that keeps retrying obscures the reason it was blocked, and the fix — a proper User-Agent, a slower cadence, deduplication, or moving off the shared service — never gets made.

Why deduplication beats every other speed-up

Rate limiting bounds requests, not rows. A file with 40,000 rows and 12,000 distinct normalised addresses needs 12,000 requests, and if 9,000 are already cached from last month it needs 3,000 — under an hour at one per second.

That ordering is worth internalising: reduce the number of questions before trying to ask them faster. It is also the only speed-up that no usage policy objects to.

Bar chart of an exponential backoff schedule capped at sixty seconds.
Uncapped exponential growth parks a pipeline for hours on a transient problem.

Edge cases or notes

  • Check the response body on a 429. Some services distinguish a per-second limit from a monthly quota, and only one of them is fixable by waiting.
  • Concurrency limits exist separately from rate limits. Four parallel workers can fail where sixteen sequential requests succeed.
  • Shared IPs share limits. Office NAT, CI runners and cloud egress addresses are often already spending the quota.
  • Retry-After can be an HTTP date, not just a number of seconds.
  • Cap the backoff. Exponential growth with no ceiling parks a pipeline for hours.
  • Add jitter when more than one job can hit the same limit.
  • Log the status code, not just "failed". 429, 403, 502 and an empty result need different responses.
  • A block can outlive the job. Test with a single request before re-launching a batch.

FAQ

What is the difference between 429 and 403?

429 is a rate limit — slow down and retry. 403 from a shared open service is usually a policy block on your IP, which retrying makes worse. Stop, fix the client, and consider self-hosting.

How fast can I geocode against the public Nominatim service?

About one request per second, sequentially, with a genuine User-Agent. Bulk geocoding is explicitly discouraged regardless of pacing.

Will threads make my batch faster?

No — they will get you blocked. The limit is on total requests per second. Deduplicate, cache, then buy capacity or self-host.

Should I honour the Retry-After header?

Yes. It tells you exactly when the server will serve you again. Cap whatever value you compute so a bad header cannot stall the pipeline for hours.

How do I recover from a block?

Stop the job, fix the cause — User-Agent, cadence, concurrency — and wait. Test with a single request before restarting a batch. The completed work should already be in the cache.

What rate limit should I set for my own instance?

Whatever your hardware sustains. Measure the achieved rate rather than assuming it; a client that sleeps one second per request but spends 0.15 s in flight is running at 0.87 requests per second.