⚙️ How-To Pipelines & automation Automation & Pipelines · step 11 Notebook By · Updated

How to Add Retries and Timeouts to an Automated GIS Job in Python

On this page

Not every failure means something is broken. A WFS server drops one request in fifty. A network share goes unavailable for nine seconds while a backup runs. A database refuses a connection because the pool was momentarily full. Run a job by hand and you shrug and press up-arrow; run it at 02:15 with nobody watching and that same nine-second blip becomes a failed pipeline and a stale layer. Retries make transient failures invisible — and, done carelessly, they turn a two-minute failure into a two-hour hang. This article covers both halves: retrying what deserves it, and putting a hard ceiling on how long anything is allowed to take.

Problem statement

An automated GIS job fails on things that would have worked if tried again. Typically:

  • Flaky network reads — a WFS or tile request fails once, and the whole run dies with it.
  • Contended files — a file on a network share is locked by another process for a few seconds.
  • Database hiccups — a PostGIS connection is refused during a failover, then works fine a second later.
  • Jobs that hang forever — a request to a dead server never returns, and the job is still "running" the next morning.
  • Retrying the wrong things — a missing file or a bad CRS is retried five times, turning an instant failure into a slow one with the same outcome.

Transient failures should be retried with backoff, permanent ones raised at once, and every external call bounded by a timeout so the job cannot hang. A real service cannot be made to fail on cue, so the examples run against a small local server that serves the ONS boundary of Rutland and misbehaves on purpose: some paths answer 503 a set number of times, one returns 404, one rate-limits, and one accepts the connection and never answers.

Quick answer

Wrap only the calls that touch something outside your process — network, database, remote filesystem — in a retry with exponential backoff and jitter. Decide what to retry with a predicate, not a broad exception type, and always pass an explicit timeout. For a remote vector file, fetch the bytes yourself — where you control both — and hand them to GeoPandas.

Four attempts with exponential backoff: 1s, 2s, 4s, then give up. exponential backoff with jitterattempt 1 → failswait 1sattempt 2 → failswait 2sattempt 3 → failswait 4sattempt 4 → raisegive up, alert08sBack off fast, add jitter, and give up decisively — then alert spatialworkflow.io
Exponential backoff with jitter: back off fast, give up decisively.
RETRY_STATUS = {408, 429, 500, 502, 503, 504}

def is_transient(exc: Exception) -> bool:
    if isinstance(exc, requests.HTTPError):
        return exc.response.status_code in RETRY_STATUS
    return isinstance(exc, (requests.ConnectionError, requests.Timeout, TimeoutError, ConnectionError,
                            BlockingIOError))

def with_retry(fn, *args, attempts=4, base_delay=0.5, retry_if=is_transient, **kwargs):
    """Call fn, retrying transient failures with exponential backoff + jitter."""
    start = time.monotonic()
    for attempt in range(1, attempts + 1):
        try:
            return fn(*args, **kwargs)
        except Exception as exc:
            if not retry_if(exc):
                raise                                            # permanent: fail now
            if attempt == attempts:
                log.error("%s: giving up after %d attempts over %.0f s: %s",
                          fn.__name__, attempts, time.monotonic() - start, exc)
                raise
            delay = base_delay * 2 ** (attempt - 1)
            delay += random.uniform(0, delay * 0.1)              # jitter
            log.warning("%s: %s; retry %d/%d in %.2f s", fn.__name__, exc, attempt, attempts - 1, delay)
            time.sleep(delay)

def read_remote(url, timeout=(3, 30)):
    response = requests.get(url, timeout=timeout)                # (connect, read) seconds
    response.raise_for_status()
    return gpd.read_file(io.BytesIO(response.content))

boundary = with_retry(read_remote, f"{BASE}/flaky/2/rutland.geojson")
print(boundary["LAD24NM"].iloc[0], "read after", hits["/flaky/2/rutland.geojson"], "requests")
WARNING read_remote: 503 Server Error: Service Unavailable for url: …/flaky/2/rutland.geojson; retry 1/3 in 0.52 s
WARNING read_remote: 503 Server Error: Service Unavailable for url: …/flaky/2/rutland.geojson; retry 2/3 in 1.02 s
Rutland read after 3 requests

The important part is not the loop — it is retry_if. Retrying everything is a bug, not a feature, and as the next section shows, the obvious exception types do not mean what they appear to.

Transient or permanent, back off, always time out

Sort failures into transient and permanent

This is the decision the whole design rests on. A transient failure might succeed if you try again in a moment: a timeout, a dropped connection, a locked file, an HTTP 503. A permanent failure will fail identically forever: a missing file, a malformed geometry, an unknown CRS, an HTTP 404. Retrying a permanent failure wastes time and hides the real error behind three copies of itself.

Retry transient failures; fail immediately on permanent ones. could a second attempt succeed?What kind of failure is this?transientRetry with backofftimeout, 503, locked filepermanentRaise immediatelymissing file, bad CRS, 404repeatedOpen the circuit5 in a row → service is downRetrying a permanent failure only delays the real error message spatialworkflow.io
Retry what a second attempt could fix; fail fast on everything else.

The trap is that exception types do not line up with that split. Here are six real failures, and what each one is an instance of:

def locked_file():
    holder = open("data/lockfile", "w")
    fcntl.flock(holder, fcntl.LOCK_EX)                       # another process holds the lock
    with open("data/lockfile", "w") as mine:
        fcntl.flock(mine, fcntl.LOCK_EX | fcntl.LOCK_NB)     # we try without waiting

cases = {
    "HTTP 503": lambda: requests.get(f"{BASE}/flaky/9/a.geojson", timeout=3).raise_for_status(),
    "HTTP 404": lambda: requests.get(f"{BASE}/missing/a.geojson", timeout=3).raise_for_status(),
    "refused": lambda: requests.get(DEAD, timeout=3),
    "read timeout": lambda: requests.get(f"{BASE}/hang/a.geojson", timeout=(3, 1)),
    "missing file": lambda: gpd.read_file("data/no_such_layer.gpkg"),
    "locked file": locked_file,
}
print(f"{'failure':13} {'exception':22} {'builtin ConnectionError/TimeoutError':>37} {'OSError':>8} {'is_transient':>13}")
for name, call in cases.items():
    try:
        call()
    except Exception as exc:
        builtin = isinstance(exc, (ConnectionError, TimeoutError))
        print(f"{name:13} {type(exc).__name__:22} {builtin!s:>37} {isinstance(exc, OSError)!s:>8} {is_transient(exc)!s:>13}")
failure       exception               builtin ConnectionError/TimeoutError  OSError  is_transient
HTTP 503      HTTPError                                              False     True          True
HTTP 404      HTTPError                                              False     True         False
refused       ConnectionError                                        False     True          True
read timeout  ReadTimeout                                            False     True          True
missing file  DataSourceError                                        False    False         False
locked file   BlockingIOError                                        False     True          True
Six real failures, whether each is a built-in ConnectionError or TimeoutError, an OSError, and whether it should be retried. exception types do not follow the transient/permanent splitbuiltin Conn/Timeout?OSError?retry?HTTP 503noyes✓retryHTTP 404noyes✕fail nowconnection refusednoyes✓retryread timeoutnoyes✓retrymissing file (GDAL)nono✕fail nowlocked filenoyes✓retryNeither type test gives the right-hand column; a predicate does spatialworkflow.io
The table above as a grid: the two type tests on the left cannot produce the decision on the right.

Three things in that table break the obvious approach:

  • requests' exceptions are not Python's. requests.ConnectionError and requests.Timeout are not the built-in ConnectionError and TimeoutError, so retry_on=(TimeoutError, ConnectionError) — the textbook tuple — retries neither a refused connection nor a timeout from requests.
  • OSError is far too wide. Every requests exception is an OSError, including the one raised for a 404, so adding OSError to "catch the network errors" retries permanent failures.
  • GDAL errors carry no type information. A missing file read through GeoPandas raises pyogrio's DataSourceError, a RuntimeError, and so does a failed HTTP read through GDAL — whatever the status code. You can only tell them apart by the message.

That is why the quick answer decides with a function: status codes for HTTP, the specific transient classes otherwise. For HTTP, retry 408, 429, 500, 502, 503 and 504; never retry 400, 401, 403 or 404. A lock held by another process raises BlockingIOError on Linux and macOS when you ask not to wait; on Windows, a file another program has open typically fails with a PermissionError (a sharing violation), so there a lock and a permissions problem share a type too.

Back off exponentially, and add jitter

Retrying immediately usually fails again — whatever was overloaded a millisecond ago still is. Doubling the wait after each attempt gives the other side time to recover, and a small random addition (jitter) stops multiple parallel workers from retrying in lockstep and re-creating the spike that caused the failure.

for attempt in range(1, 5):
    delay = 1.0 * 2 ** (attempt - 1)                 # 1 s, 2 s, 4 s, 8 s
    print(f"attempt {attempt}: wait {delay:.0f} s + up to {delay * 0.1:.1f} s of jitter")
attempt 1: wait 1 s + up to 0.1 s of jitter
attempt 2: wait 2 s + up to 0.2 s of jitter
attempt 3: wait 4 s + up to 0.4 s of jitter
attempt 4: wait 8 s + up to 0.8 s of jitter

Cap the total: three waits of 1, 2 and 4 seconds — about seven seconds over four attempts — is right for most GIS jobs. If the service is genuinely down, more attempts only delay the alert.

Always pass a timeout

An unbounded call is a worse failure than a crash, because nothing tells you it is stuck. requests never times out unless told to; pair a short connect timeout with a longer read timeout for the large payloads GIS work involves.

start = time.monotonic()
try:
    requests.get(f"{BASE}/hang/b.geojson", timeout=(3, 2))    # 3 s to connect, 2 s between bytes
except requests.Timeout as exc:
    print(type(exc).__name__, "after", round(time.monotonic() - start), "s")
ReadTimeout after 2 s

The read timeout is the time allowed between bytes, not for the whole download: a server that trickles one byte a second never trips it. Where a total deadline matters, enforce it outside the call, as example 3 does.

The database needs the same bounds — one on connecting and one on any single statement:

engine = create_engine("postgresql+psycopg://user@localhost/gis",
                       connect_args={"connect_timeout": 10, "options": "-c statement_timeout=2000"})
start = time.monotonic()
try:
    with engine.connect() as conn:
        conn.execute(text("SELECT pg_sleep(30)"))              # a query that would run 30 s
except Exception as exc:
    print(type(exc).__name__, "after", round(time.monotonic() - start), "s:", str(exc.orig).splitlines()[0])

start = time.monotonic()
try:
    create_engine("postgresql+psycopg://user@10.255.255.1/gis",   # an address that never answers
                  connect_args={"connect_timeout": 3}).connect()
except Exception as exc:
    print(type(exc).__name__, "after", round(time.monotonic() - start), "s")
OperationalError after 2 s: canceling statement due to statement timeout
OperationalError after 3 s

PostgreSQL cancelled the query at two seconds, and the connection to an address that never answers gave up at three. Without connect_timeout, that connection waits on the operating system's TCP timeout, which is minutes.

Know which library is doing the HTTP

gpd.read_file() with a URL does not simply hand it to GDAL. GeoPandas first probes the server with Python's urllib — which has no timeout — and then either passes the URL on to GDAL or downloads the whole file itself with urllib, again with no timeout. So GDAL's HTTP settings may not apply at all:

pyogrio.set_gdal_config_options({"GDAL_HTTP_MAX_RETRY": "3", "GDAL_HTTP_RETRY_DELAY": "1",
                                 "GDAL_HTTP_TIMEOUT": "3"})
for label, url in (("plain URL", f"{BASE}/flaky/2/c.geojson"), ("/vsicurl/ URL", f"/vsicurl/{BASE}/flaky/2/d.geojson")):
    start = time.monotonic()
    try:
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")                     # GDAL reports each retry as a warning
            layer = gpd.read_file(url)
        print(f"{label:14} read {len(layer)} feature in {time.monotonic() - start:.1f} s")
    except Exception as exc:
        print(f"{label:14} {type(exc).__name__}: {exc}")
plain URL      HTTPError: HTTP Error 503: Service Unavailable
/vsicurl/ URL  read 1 feature in 3.5 s

The plain URL failed on the first 503: that error came from urllib, and GDAL's retry setting never saw the request. Prefixed with /vsicurl/, GDAL itself made the requests, honoured GDAL_HTTP_MAX_RETRY, and read the layer after two retries. Use /vsicurl/ when you want GDAL's settings to govern the read — and set them with pyogrio.set_gdal_config_options() or in the environment before Python starts.

The two ways to pass a URL to gpd.read_file, and which library makes the HTTP requests in each. read_file(url): who makes the request?gpd.read_file("https://host/layer.geojson")gpd.read_file("/vsicurl/https://host/layer.geojson")①②①GeoPandas probes with urllib first, with no timeout. GDAL_HTTP_* settingsnever see that request; a 503 fails at once.②GDAL makes every request: GDAL_HTTP_TIMEOUT applies per request andGDAL_HTTP_MAX_RETRY retries 503s and timeouts.Only the /vsicurl/ route is governed by GDAL’s settings from the first request spatialworkflow.io
Why the plain URL failed on the first 503 while the /vsicurl/ read retried.

GDAL's own retries stack with its timeout. Against the server that never answers:

start = time.monotonic()
try:
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        gpd.read_file(f"/vsicurl/{BASE}/hang/e.geojson")
except Exception as exc:
    print(f"{type(exc).__name__} after {time.monotonic() - start:.0f} s and {hits['/hang/e.geojson']} requests: "
          f"{str(exc)[:60]}")
pyogrio.set_gdal_config_options({"GDAL_HTTP_MAX_RETRY": None, "GDAL_HTTP_RETRY_DELAY": None, "GDAL_HTTP_TIMEOUT": None})
DataSourceError after 64 s and 12 requests: CURL error: Operation timed out after 3002 milliseconds with

A three-second timeout, and the read took about a minute: GDAL retried the timed-out requests, and /vsicurl/ makes several requests to open one file. GDAL_HTTP_TIMEOUT bounds each request, not the read. And if you wrap that read in your own three-attempt retry as well, the worst case multiplies again.

Wrap the call, not the pipeline

Retry the smallest thing that can fail on its own. Retrying a whole stage re-runs work that already succeeded and, if that stage writes files, can leave half-written output behind. The right unit is one external call:

def extract(url):
    layer = with_retry(read_remote, url)                           # just the read is retried
    return layer.to_crs(27700)                                     # never retried

print(round(extract(f"{BASE}/flaky/1/f.geojson").area.iloc[0] / 1e6, 1), "km²")
WARNING read_remote: 503 Server Error: Service Unavailable for url: …/flaky/1/f.geojson; retry 1/3 in 0.53 s
393.7 km²

Make retried operations safe to repeat

A retry runs the operation again, so anything with a side effect must be safe to repeat. Write to a temporary file and rename it into place once complete: a rename is atomic on the same filesystem, so a reader never sees a partial file and a retry never appends to a corrupt one.

def write_atomic(gdf, out_path: Path, driver="GPKG"):
    tmp = out_path.with_name(f"{out_path.stem}-partial{out_path.suffix}")   # same folder, same extension
    gdf.to_file(tmp, driver=driver, layer=out_path.stem)                     # name the layer for the final file
    tmp.replace(out_path)          # atomic on the same filesystem

Path("output").mkdir(exist_ok=True)
write_atomic(boundary, Path("output/rutland.gpkg"))
print(sorted(p.name for p in Path("output").iterdir()), gpd.read_file("output/rutland.gpkg").crs)
['rutland.gpkg'] EPSG:4326

Two details matter. Keep the temporary file's extension, because GDAL picks a driver from it. And name the layer: without layer=, GDAL names it after the file, so the renamed GeoPackage would contain a layer called rutland-partial — and a name like .rutland.tmp is refused outright ("The layer name may not contain special characters or spaces"), as the first version of this function found.

Use a library when the rules get complicated

Hand-rolled retry is fine for one or two call sites. Past that, tenacity expresses the policy declaratively and is well tested — and it takes a predicate too:

from tenacity import (RetryError, before_sleep_log, retry, retry_if_exception, stop_after_attempt,
                      wait_exponential_jitter)

@retry(stop=stop_after_attempt(4), wait=wait_exponential_jitter(initial=0.5, max=5),
       retry=retry_if_exception(is_transient), before_sleep=before_sleep_log(log, logging.WARNING),
       reraise=True)
def fetch_layer(url):
    return read_remote(url)

print(len(fetch_layer(f"{BASE}/flaky/1/g.geojson")), "feature")

without_reraise = retry(stop=stop_after_attempt(2), wait=wait_exponential_jitter(initial=0.1),
                        retry=retry_if_exception(is_transient))(read_remote)
try:
    without_reraise(f"{BASE}/flaky/9/h.geojson")
except RetryError as exc:
    print("without reraise=True:", type(exc).__name__, "| the real error:",
          str(exc.last_attempt.exception()).replace(BASE, "…"))
WARNING Retrying __guide__.fetch_layer in 0.572 seconds as it raised HTTPError: 503 Server Error: Service Unavailable for url: …/flaky/1/g.geojson.
1 feature
without reraise=True: RetryError | the real error: 503 Server Error: Service Unavailable for url: …/flaky/9/h.geojson

reraise=True matters: without it, tenacity raises its own RetryError and your caller's except requests.HTTPError never fires.

Give up loudly

The last attempt should produce a clear, actionable failure — which call, how many attempts, over what period, and the underlying error. A retry policy that ends in a bare traceback has only made the failure later, not clearer:

try:
    with_retry(read_remote, f"{BASE}/flaky/9/i.geojson", attempts=3, base_delay=0.2)
except requests.HTTPError:
    pass
WARNING read_remote: 503 Server Error: Service Unavailable for url: …/flaky/9/i.geojson; retry 1/2 in 0.20 s
WARNING read_remote: 503 Server Error: Service Unavailable for url: …/flaky/9/i.geojson; retry 2/2 in 0.42 s
ERROR read_remote: giving up after 3 attempts over 1 s: 503 Server Error: Service Unavailable for url: …/flaky/9/i.geojson

Retrying a deliberately flaky map service

How to bound each kind of call, and what happens if you do not. none of these has a deadline by defaultthe callhow you bound itif you do notrequests.gettimeout=(3, 30)waits foreverSQLAlchemy + psycopgconnect_timeout, statement_timeoutOS timeout / noneread_file(url)requests, or /vsicurl/ + GDAL_HTTP_*urllib: no timeoutGDAL settings only reach /vsicurl/ reads; a plain URL goes through urllib firstAn unbounded call is worse than a crash, because nothing tells you it is stuck spatialworkflow.io
A retry around an unbounded call never fires, because the call never returns.

The data used on this page

The Rutland boundary from the ONS December 2024 local authority boundaries (OGL), served by a local stand-in for a flaky map service. /flaky/N/… answers 503 to its first N requests, /ratelimit/… answers 429 once with Retry-After: 2, /missing/… answers 404, and /hang/… accepts the connection and sends nothing for 20 seconds. DEAD is a port with nothing listening. The PostGIS examples start a throwaway container, as other guides on this site do.

import fcntl
import io
import logging
import os
import random
import re
import shutil
import socket
import sys
import threading
import time
import warnings
from collections import Counter
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path

import geopandas as gpd
import pyogrio
import requests
from sqlalchemy import create_engine, text

Path("data").mkdir(exist_ok=True)
shutil.rmtree("output", ignore_errors=True)
if not Path("data/rutland_boundary.geojson").exists():
    Path("data/rutland_boundary.geojson").write_bytes(requests.get(
        "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/"
        "Local_Authority_Districts_December_2024_Boundaries_UK_BFC/FeatureServer/0/query",
        params={"where": "LAD24NM='Rutland'", "outFields": "LAD24NM", "outSR": 4326, "f": "geojson"}, timeout=120).content)
BODY = Path("data/rutland_boundary.geojson").read_bytes()
hits = Counter()

class FlakyService(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    def log_message(self, *args):
        pass
    def reply(self, status, body=b"", headers=()):
        self.send_response(status)
        for key, value in headers:
            self.send_header(key, value)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        if self.command == "GET":
            self.wfile.write(body)
    def respond(self):
        hits[self.path] += 1
        kind, *rest = self.path.strip("/").split("/")
        if kind == "flaky" and hits[self.path] <= int(rest[0]):
            return self.reply(503)
        if kind == "ratelimit" and hits[self.path] == 1:
            return self.reply(429, headers=[("Retry-After", "2")])
        if kind == "missing":
            return self.reply(404)
        if kind == "hang":
            time.sleep(20)
            self.close_connection = True
            return
        found = re.match(r"bytes=(\d+)-(\d*)", self.headers.get("Range", ""))     # serve ranges, like S3 or nginx
        if found:
            first = int(found.group(1))
            last = min(int(found.group(2) or len(BODY) - 1), len(BODY) - 1)
            return self.reply(206, BODY[first:last + 1], [("Content-Range", f"bytes {first}-{last}/{len(BODY)}"),
                                                          ("Accept-Ranges", "bytes")])
        return self.reply(200, BODY, [("Content-Type", "application/geo+json"), ("Accept-Ranges", "bytes")])
    do_GET = do_HEAD = respond

server = ThreadingHTTPServer(("127.0.0.1", 0), FlakyService)
server.daemon_threads = True
threading.Thread(target=server.serve_forever, daemon=True).start()
BASE = f"http://127.0.0.1:{server.server_address[1]}"
with socket.socket() as probe:
    probe.bind(("127.0.0.1", 0))
    DEAD = f"http://127.0.0.1:{probe.getsockname()[1]}/layer.geojson"            # nothing listens here

class PrintHandler(logging.Handler):                     # log lines appear with the page's output
    def emit(self, record):
        print(re.sub(r"http://127\.0\.0\.1:\d+|host='127\.0\.0\.1', port=\d+", "…", self.format(record)))

handler = PrintHandler()
handler.setFormatter(logging.Formatter("%(levelname)s %(message)s"))
logging.basicConfig(handlers=[handler], force=True)
log = logging.getLogger("retry")
random.seed(7)                                                   # makes this page's jitter repeatable

os.environ.update(PGPORT="55433", PGPASSWORD="guide")
if os.system("docker inspect guide-postgis >/dev/null 2>&1") != 0:
    os.system("docker run -d --name guide-postgis -e POSTGRES_USER=user -e POSTGRES_PASSWORD=guide "
              "-e POSTGRES_DB=gis -p 127.0.0.1:55433:5432 postgis/postgis:16-3.4 >/dev/null")
    time.sleep(10)
print("serving", len(BODY), "bytes of", gpd.read_file(io.BytesIO(BODY))["LAD24NM"].iloc[0])
serving 482303 bytes of Rutland

Example 1: A reusable decorator

Same logic as the function form, applied at the definition site so every call is protected.

import functools

def retrying(attempts=4, base_delay=0.5, retry_if=is_transient):
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            return with_retry(fn, *args, attempts=attempts, base_delay=base_delay, retry_if=retry_if, **kwargs)
        return wrapper
    return decorator

@retrying(attempts=5, base_delay=0.2)
def fetch_features(url):
    return read_remote(url)

print(len(fetch_features(f"{BASE}/flaky/3/j.geojson")), "feature after", hits["/flaky/3/j.geojson"], "requests")
WARNING fetch_features: 503 Server Error: Service Unavailable for url: …/flaky/3/j.geojson; retry 1/4 in 0.20 s
WARNING fetch_features: 503 Server Error: Service Unavailable for url: …/flaky/3/j.geojson; retry 2/4 in 0.42 s
WARNING fetch_features: 503 Server Error: Service Unavailable for url: …/flaky/3/j.geojson; retry 3/4 in 0.81 s
1 feature after 4 requests

Example 2: Retrying HTTP by status code, honouring Retry-After

requests does not raise on a 503 until you ask it to, so decide by status — and on a 429, wait as long as the server says.

def get_with_retry(url, attempts=4, timeout=(3, 30), **kwargs):
    for attempt in range(1, attempts + 1):
        try:
            resp = requests.get(url, timeout=timeout, **kwargs)
            if resp.status_code in RETRY_STATUS and attempt < attempts:
                wait = float(resp.headers.get("Retry-After", 2 ** (attempt - 1)))
                print(f"  {resp.status_code}: waiting {wait:.0f} s")
                time.sleep(wait + random.uniform(0, 0.5))
                continue
            resp.raise_for_status()          # 4xx → raises, not retried
            return resp
        except (requests.Timeout, requests.ConnectionError):
            if attempt == attempts:
                raise
            time.sleep(2 ** (attempt - 1) + random.uniform(0, 0.5))

print(get_with_retry(f"{BASE}/ratelimit/k.geojson").status_code)
try:
    get_with_retry(f"{BASE}/missing/l.geojson")
except requests.HTTPError as exc:
    print("404 raised at once:", exc.response.status_code, "after", hits["/missing/l.geojson"], "request")
  429: waiting 2 s
200
404 raised at once: 404 after 1 request

Honouring Retry-After on a 429 is the difference between being rate-limited and being blocked.

Example 3: A hard deadline around code that does not accept one

Some calls — gpd.read_file() on a plain URL, a GDAL driver, a long geometry operation — offer no deadline for the whole operation. Run them in a separate process and kill the process if it overruns:

from concurrent.futures import ProcessPoolExecutor, TimeoutError as FutureTimeout

Path("deadline_worker.py").write_text('''import geopandas as gpd

def read_layer(url):
    return len(gpd.read_file(url))
''')                             # a process pool needs its worker function in an importable module
from deadline_worker import read_layer

def read_with_deadline(url, seconds):
    pool = ProcessPoolExecutor(max_workers=1)
    future = pool.submit(read_layer, url)
    try:
        return future.result(timeout=seconds)
    except FutureTimeout:
        for proc in pool._processes.values():                 # private, but there is no public kill
            proc.terminate()
        raise TimeoutError(f"reading {url} exceeded {seconds} s") from None
    finally:
        pool.shutdown(wait=False, cancel_futures=True)

start = time.monotonic()
try:
    read_with_deadline(f"{BASE}/hang/m.geojson", seconds=4)
except TimeoutError as exc:
    print(exc.args[0].replace(BASE, "…"), "| returned after", round(time.monotonic() - start), "s")
print("and a healthy read:", read_with_deadline(f"{BASE}/ok/n.geojson", seconds=30), "feature")
reading …/hang/m.geojson exceeded 4 s | returned after 4 s
and a healthy read: 1 feature

The worker function lives in a module because the pool's processes import it by name: a function defined in a notebook cell or in __main__ of an interactive session cannot be sent to a process started with spawn or forkserver — the default on Linux from Python 3.14, and on Windows and macOS already. A thread cannot be killed in Python, so a separate process is the only reliable way to enforce a deadline on blocking C code — or on the urllib call inside read_file, which would otherwise have waited out the server's full 20 seconds, twice. The TimeoutError it raises is the built-in one, so is_transient retries it.

Example 4: Per-file retries inside a batch loop

In a batch job, a retry belongs around each file, and a final failure should skip that file rather than end the run.

summary = []
for name in ("flaky/1/p.geojson", "missing/q.geojson", "ok/r.geojson", "flaky/9/s.geojson"):
    try:
        layer = with_retry(read_remote, f"{BASE}/{name}", attempts=3, base_delay=0.2)
        write_atomic(layer, Path("output") / name.replace("/", "_").replace(".geojson", ".gpkg"))
        summary.append((name, "ok"))
    except Exception as exc:
        summary.append((name, f"failed: {type(exc).__name__} {getattr(getattr(exc, 'response', None), 'status_code', '')}"))
for row in summary:
    print(*row)
WARNING read_remote: 503 Server Error: Service Unavailable for url: …/flaky/1/p.geojson; retry 1/2 in 0.21 s
WARNING read_remote: 503 Server Error: Service Unavailable for url: …/flaky/9/s.geojson; retry 1/2 in 0.22 s
WARNING read_remote: 503 Server Error: Service Unavailable for url: …/flaky/9/s.geojson; retry 2/2 in 0.40 s
ERROR read_remote: giving up after 3 attempts over 1 s: 503 Server Error: Service Unavailable for url: …/flaky/9/s.geojson
flaky/1/p.geojson ok
missing/q.geojson failed: HTTPError 404
ok/r.geojson ok
flaky/9/s.geojson failed: HTTPError 503

The retry handled the blip; the try/except handled the file that was genuinely missing and the one that stayed down. The 404 was reported without a single retry. See batch process a folder of GIS files.

Example 5: A circuit breaker for a service that is simply down

When a service is down, retrying every file wastes the whole run. Stop after enough consecutive failures:

class CircuitBreaker:
    def __init__(self, threshold=3):
        self.threshold, self.failures = threshold, 0

    def call(self, fn, *args, **kwargs):
        if self.failures >= self.threshold:
            raise RuntimeError(f"circuit open: {self.failures} consecutive failures — service is down")
        try:
            result = fn(*args, **kwargs)
        except Exception:
            self.failures += 1
            raise
        self.failures = 0
        return result

breaker = CircuitBreaker(threshold=3)
start = time.monotonic()
for tile in range(10):
    try:
        breaker.call(with_retry, read_remote, DEAD, attempts=2, base_delay=0.2)
    except RuntimeError as exc:
        print(f"tile {tile}: {exc}")
        break
    except requests.ConnectionError:
        print(f"tile {tile}: connection refused (after retries)")
print(f"stopped after {time.monotonic() - start:.1f} s instead of trying all 10 tiles")
WARNING read_remote: HTTPConnectionPool(…): Max retries exceeded with url: /layer.geojson (Caused by NewConnectionError("HTTPConnection(…): Failed to establish a new connection: [Errno 111] Connection refused")); retry 1/1 in 0.20 s
ERROR read_remote: giving up after 2 attempts over 0 s: HTTPConnectionPool(…): Max retries exceeded with url: /layer.geojson (Caused by NewConnectionError("HTTPConnection(…): Failed to establish a new connection: [Errno 111] Connection refused"))
tile 0: connection refused (after retries)
WARNING read_remote: HTTPConnectionPool(…): Max retries exceeded with url: /layer.geojson (Caused by NewConnectionError("HTTPConnection(…): Failed to establish a new connection: [Errno 111] Connection refused")); retry 1/1 in 0.21 s
ERROR read_remote: giving up after 2 attempts over 0 s: HTTPConnectionPool(…): Max retries exceeded with url: /layer.geojson (Caused by NewConnectionError("HTTPConnection(…): Failed to establish a new connection: [Errno 111] Connection refused"))
tile 1: connection refused (after retries)
WARNING read_remote: HTTPConnectionPool(…): Max retries exceeded with url: /layer.geojson (Caused by NewConnectionError("HTTPConnection(…): Failed to establish a new connection: [Errno 111] Connection refused")); retry 1/1 in 0.22 s
ERROR read_remote: giving up after 2 attempts over 0 s: HTTPConnectionPool(…): Max retries exceeded with url: /layer.geojson (Caused by NewConnectionError("HTTPConnection(…): Failed to establish a new connection: [Errno 111] Connection refused"))
tile 2: connection refused (after retries)
tile 3: circuit open: 3 consecutive failures — service is down
stopped after 0.6 s instead of trying all 10 tiles

Retries bet that the world will change

Retries are a bet that the world will be different in a second. That bet pays off for anything crossing a network or a filesystem boundary, and never pays off for logic errors or bad data. Splitting failures into transient and permanent is therefore not a refinement — it is the design. A job that retries a missing file four times takes seven extra seconds to deliver exactly the same message, and it teaches whoever reads the log to distrust retry warnings entirely. The split has to be made on what the error means, because, as the table above showed, the exception classes do not make it for you.

Exponential backoff exists because the most common cause of a transient failure is load, and immediate retries add load. Doubling the delay gives the other side room; jitter prevents synchronised herds of retries from re-creating the spike. Both matter more as soon as you run work in parallel, where a dozen workers can hammer a struggling server simultaneously.

An unbounded call hangs indefinitely; a bounded one fails at a known deadline. every call that leaves the process gets a deadlineNo timeout✕one dead server hangs the job✕still "running" next morning✕blocks the next scheduled run✕no alert — it never failedtimeout=(5, 120)✓fails after a known deadline✓the failure is retryable✓the run finishes either way✓alert fires with a reasonA retry policy without timeouts is not a safety net spatialworkflow.io
A retry without a timeout is not a safety net — one hung call holds the job open forever.

Timeouts are the more important half of this article, and the one more often skipped. A retry policy without timeouts is not a safety net — a single hung connection can hold a job open indefinitely, past the next scheduled run, until someone notices the following week. The rule is absolute: every call that leaves the process gets a deadline, and you must know which library is actually making the call. gpd.read_file() on a URL made its first request with urllib, which no GDAL setting reaches. Where nothing offers a deadline, put the work in a subprocess you can kill, as in Example 3.

Idempotence makes retries safe. Because a retried operation may run more than once, any side effect must tolerate repetition: write to a temporary file and rename, use a transaction and commit once, or derive the output name from the input so a repeat overwrites its own work rather than appending. Get this wrong and a retry produces a half-written GeoPackage or a table with duplicated rows — a worse outcome than the original failure. The same reasoning underpins scheduling, where a job may be run twice by a scheduler that catches up after downtime.

Finally, retries hide problems by design, so make sure they are visible. Log every retry at warning level with the reason, and count them. A job that quietly retries twice on every run is telling you something about the service it depends on, and that information is worth more than the seconds the retries saved.

Half-written files, workers and time budgets

Three ways a retry loop makes things worse rather than better. what a naive retry actually doeswhat the retry doeswhat to doSecond attempt meets half a fileWrite to .tmp, then Path.replace()workers x attempts hit one serviceJitter, a circuit breaker, fewerworkersrequests returns normally on a 500raise_for_status(), then catch itPath.replace() is atomic within a filesystem: a reader sees the old file or the new one spatialworkflow.io
The third is the one that makes a retry loop do nothing at all.

Retrying a partially written file

If the first attempt failed halfway through writing, the second may append to or trip over the debris. Always write to a temporary path and replace() it into place. Path.replace() is atomic within one filesystem, so a reader sees either the old file or the new one, never a half.

Retries inside parallel workers

Each worker retries independently, so a service already under strain receives workers × attempts requests. Jitter helps; a shared circuit breaker helps more. When throughput matters more than latency, reduce the worker count rather than the retry count.

requests does not raise on HTTP errors

requests.get() returns normally for a 500 or a 503, so a naïve try/except around it retries nothing. Either check resp.status_code explicitly or call resp.raise_for_status() — and remember that turns 4xx into exceptions too, which you should not retry.

Timeouts through GDAL

gpd.read_file() on a plain URL makes its first request with Python's urllib, which has no timeout and ignores every GDAL setting. Prefix the URL with /vsicurl/ to have GDAL make the requests, then configure GDAL_HTTP_TIMEOUT, GDAL_HTTP_CONNECTTIMEOUT, GDAL_HTTP_MAX_RETRY and GDAL_HTTP_RETRY_DELAY with pyogrio.set_gdal_config_options() or in the environment before Python starts. The timeout is per request and GDAL retries timed-out requests, so a three-second timeout took about a minute to give up on a server that never answered.

Retrying a database write

A retried INSERT can duplicate rows if the first attempt actually succeeded and only the acknowledgement was lost. Wrap writes in an explicit transaction, or make them idempotent with a unique key and an upsert. This is the one case where a retry can silently corrupt data rather than merely waste time.

Total time budget

Four attempts at 1, 2, 4, and 8 seconds is fifteen seconds — fine. The same policy applied per file across 500 files, with a service that is down, is over two hours of sleeping. Bound the job, not just each call: track elapsed time and abort the run when a budget is exhausted.

FAQ

Which errors should I retry in a GIS job?

Only those a second attempt could plausibly fix: timeouts, connection resets, temporarily locked files, and HTTP 429/5xx responses. Decide with a function rather than an exception tuple: requests.Timeout is not the built-in TimeoutError, every requests error — a 404 included — is an OSError, and GDAL raises the same RuntimeError subclass for a missing file and a failed download. Never retry missing files, permission errors, invalid CRS codes, malformed geometry, or HTTP 4xx — they will fail identically every time, and retrying them delays the real error message.

How many retries and how long should I wait?

Three or four attempts with exponential backoff starting at one second covers nearly all transient failures — roughly seven to fifteen seconds total. Beyond that you are usually waiting on something genuinely down, where failing quickly and alerting is more useful than continuing to wait.

What is jitter and do I need it?

Jitter is a small random addition to each backoff delay. It matters when several workers fail at the same moment: without it they all retry at exactly the same instant and re-create the overload. With parallel processing or a shared service, add it; for a single-threaded job it is harmless insurance.

How do I put a timeout on gpd.read_file() over HTTP?

Not through GDAL settings alone: on a plain URL, GeoPandas makes its first request with urllib, which has no timeout. Either download the bytes with requests.get(url, timeout=(3, 30)) and pass io.BytesIO(response.content) to read_file, or prefix the URL with /vsicurl/ so GDAL makes every request and GDAL_HTTP_TIMEOUT applies — per request, with GDAL's own retries on top. For a hard deadline on the whole read, run it in a separate process and terminate it if it overruns.

Is it safe to retry a write?

Only if the write is idempotent. Write to a temporary file and replace() it into the final path so a repeat cannot leave partial output. For databases, use a transaction or an upsert with a unique key; a plain retried INSERT can duplicate rows when the first attempt succeeded but its acknowledgement was lost.

Should I retry the whole pipeline stage or just the failing call?

Just the call. Retrying a stage repeats work that already succeeded, can double side effects, and makes the log harder to read. Wrap the single external operation — one HTTP request, one file read, one query — and let everything around it stay simple.

How do I stop retries from hiding a service that is down?

Log every retry at warning level and count them, so a rising retry rate is visible before it becomes an outage. Add a circuit breaker that stops trying after several consecutive failures, and make the final failure message say how many attempts were made over what period. Then alert on it — a job that silently retried for an hour tells nobody anything.