How to Add Retries and Timeouts to an Automated GIS Job in Python

Not every failure means something is broken. A WFS server drops one request in fifty. A network share goes unavailable for nine seconds while a backup runs. A database refuses a connection because the pool was momentarily full. Run a job by hand and you shrug and press up-arrow; run it at 02:15 with nobody watching and that same nine-second blip becomes a failed pipeline and a stale layer. Retries make transient failures invisible — and, done carelessly, they turn a two-minute failure into a two-hour hang. This article covers both halves: retrying what deserves it, and putting a hard ceiling on how long anything is allowed to take.

Problem statement

An automated GIS job fails on things that would have worked if tried again. Typically:

Flaky network reads — a WFS or tile request fails once, and the whole run dies with it.
Contended files — a shapefile on a network share is locked by another process for a few seconds.
Database hiccups — a PostGIS connection is refused during a failover, then works fine a second later.
Jobs that hang forever — a request to a dead server never returns, and the job is still "running" the next morning.
Retrying the wrong things — a missing file or a bad CRS is retried five times, turning an instant failure into a slow one with the same outcome.

The goal: transient failures retried with sensible backoff, permanent failures raised immediately, and every external call bounded by a timeout so the job can never hang indefinitely.

Quick answer

Wrap only the calls that touch something outside your process — network, database, remote filesystem — in a retry with exponential backoff and jitter. Retry only exception types that can plausibly succeed on a second attempt, and always pass an explicit timeout.

Four attempts with exponential backoff: one second, two seconds, four seconds, then failure. — Exponential backoff with jitter: back off fast, give up decisively.

import logging, random, time

log = logging.getLogger("retry")

TRANSIENT = (TimeoutError, ConnectionError, OSError)

def with_retry(fn, *args, attempts=4, base_delay=1.0, retry_on=TRANSIENT, **kwargs):
    """Call fn, retrying transient failures with exponential backoff + jitter."""
    for attempt in range(1, attempts + 1):
        try:
            return fn(*args, **kwargs)
        except retry_on as exc:
            if attempt == attempts:
                log.error("%s failed after %d attempts: %s", fn.__name__, attempts, exc)
                raise
            delay = base_delay * 2 ** (attempt - 1)
            delay += random.uniform(0, delay * 0.1)      # jitter
            log.warning("%s failed (%s); retry %d/%d in %.1fs",
                        fn.__name__, exc, attempt, attempts - 1, delay)
            time.sleep(delay)

gdf = with_retry(gpd.read_file, "https://example.org/wfs?...", attempts=4)

The important part is not the loop — it is retry_on. Retrying everything is a bug, not a feature.

Step-by-step solution

Sort failures into transient and permanent

This is the decision the whole design rests on. A transient failure might succeed if you try again in a moment: a timeout, a dropped connection, a locked file, an HTTP 503. A permanent failure will fail identically forever: a missing file, a malformed geometry, an unknown CRS, an HTTP 404. Retrying a permanent failure wastes time and hides the real error behind three copies of itself.

A decision split: timeouts and connection errors are retried, while missing files and bad data fail immediately. — Retry what a second attempt could fix; fail fast on everything else.

TRANSIENT = (
    TimeoutError,
    ConnectionError,          # includes ConnectionResetError, BrokenPipeError
    BlockingIOError,          # file locked by another process
)

PERMANENT = (
    FileNotFoundError,
    PermissionError,
    ValueError,               # bad CRS, bad geometry, bad argument
)

For HTTP work the distinction is by status code, not exception type: retry 408, 429, 500, 502, 503, and 504; never retry 400, 401, 403, or 404.

Back off exponentially, and add jitter

Retrying immediately usually fails again — whatever was overloaded a millisecond ago still is. Doubling the wait after each attempt gives the other side time to recover, and a small random addition (jitter) stops multiple parallel workers from retrying in lockstep and re-creating the spike that caused the failure.

delay = base_delay * 2 ** (attempt - 1)      # 1s, 2s, 4s, 8s
delay += random.uniform(0, delay * 0.1)      # ±10% jitter

Cap the total: four attempts over roughly seven seconds is right for most GIS jobs. If the service is genuinely down, more attempts only delay the alert.

Always pass a timeout

An unbounded call is a worse failure than a crash, because nothing tells you it is stuck. Every network library takes a timeout; use it, and pair a short connect timeout with a longer read timeout for the large payloads GIS work involves.

import requests

resp = requests.get(url, timeout=(5, 120))    # 5s to connect, 120s to read
resp.raise_for_status()

# PostGIS: bound the connection and any single statement
from sqlalchemy import create_engine

engine = create_engine(
    "postgresql+psycopg://user:pw@host/gis",
    connect_args={"connect_timeout": 10, "options": "-c statement_timeout=300000"},
)

Note that gpd.read_file() on a remote URL goes through GDAL, which does not take a Python timeout argument — configure it through GDAL's own environment settings:

import os
os.environ["GDAL_HTTP_TIMEOUT"] = "120"        # seconds
os.environ["GDAL_HTTP_CONNECTTIMEOUT"] = "10"
os.environ["GDAL_HTTP_MAX_RETRY"] = "3"
os.environ["GDAL_HTTP_RETRY_DELAY"] = "2"

Wrap the call, not the pipeline

Retry the smallest thing that can fail on its own. Retrying a whole stage re-runs work that already succeeded and, if that stage writes files, can leave half-written output behind. The right unit is one external call.

def extract(cfg):
    gdf = with_retry(gpd.read_file, cfg["wfs_url"], attempts=4)   # ← just the read
    return clean_columns(gdf)                                     # never retried

Make retried operations safe to repeat

A retry runs the operation again, so anything with a side effect must be safe to repeat. Write to a temporary file and rename it into place once complete: a rename is atomic on the same filesystem, so a reader never sees a partial file and a retry never appends to a corrupt one.

from pathlib import Path

def write_atomic(gdf, out_path: Path, driver="GPKG"):
    tmp = out_path.with_suffix(out_path.suffix + ".tmp")
    gdf.to_file(tmp, driver=driver)
    tmp.replace(out_path)          # atomic on the same filesystem

Use a library when the rules get complicated

Hand-rolled retry is fine for one or two call sites. Past that, tenacity expresses the policy declaratively and is well tested.

from tenacity import (retry, stop_after_attempt, wait_exponential_jitter,
                      retry_if_exception_type, before_sleep_log)
import logging

log = logging.getLogger("wfs")

@retry(
    stop=stop_after_attempt(4),
    wait=wait_exponential_jitter(initial=1, max=30),
    retry=retry_if_exception_type((TimeoutError, ConnectionError)),
    before_sleep=before_sleep_log(log, logging.WARNING),
    reraise=True,
)
def fetch_layer(url):
    return gpd.read_file(url)

reraise=True matters: without it, tenacity raises its own RetryError and your caller loses the original exception.

Give up loudly

The last attempt should produce a clear, actionable failure — which service, how many attempts, over what period, and the underlying error. A retry policy that ends in a bare traceback has only made the failure later, not clearer.

except retry_on as exc:
    if attempt == attempts:
        log.error("giving up on %s after %d attempts over %.1fs: %s",
                  fn.__name__, attempts, elapsed, exc)
        raise

Code examples

Example 1: A reusable decorator

Same logic as the function form, applied at the definition site so every call is protected.

import functools, logging, random, time

log = logging.getLogger("retry")

def retry(attempts=4, base_delay=1.0, exceptions=(TimeoutError, ConnectionError)):
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(1, attempts + 1):
                try:
                    return fn(*args, **kwargs)
                except exceptions as exc:
                    if attempt == attempts:
                        raise
                    delay = base_delay * 2 ** (attempt - 1)
                    delay += random.uniform(0, delay * 0.1)
                    log.warning("%s: %s — retry %d/%d in %.1fs",
                                fn.__name__, exc, attempt, attempts - 1, delay)
                    time.sleep(delay)
        return wrapper
    return decorator

@retry(attempts=5, base_delay=2)
def fetch_features(url, bbox):
    return gpd.read_file(url, bbox=bbox)

Example 2: Retrying HTTP by status code

requests does not raise on a 503 until you ask it to, so decide by status.

import requests, time, random

RETRY_STATUS = {408, 429, 500, 502, 503, 504}

def get_with_retry(url, attempts=4, timeout=(5, 120), **kwargs):
    for attempt in range(1, attempts + 1):
        try:
            resp = requests.get(url, timeout=timeout, **kwargs)
            if resp.status_code in RETRY_STATUS and attempt < attempts:
                wait = float(resp.headers.get("Retry-After", 2 ** (attempt - 1)))
                time.sleep(wait + random.uniform(0, 0.5))
                continue
            resp.raise_for_status()          # 4xx → raises, not retried
            return resp
        except (requests.Timeout, requests.ConnectionError):
            if attempt == attempts:
                raise
            time.sleep(2 ** (attempt - 1) + random.uniform(0, 0.5))

Honouring Retry-After on a 429 is the difference between being rate-limited and being blocked.

Example 3: A hard timeout around code that does not accept one

Some calls — a GDAL driver, a long geometry operation — offer no timeout parameter. Run them in a separate process and kill the process if it overruns.

from concurrent.futures import ProcessPoolExecutor, TimeoutError as FutureTimeout

def read_layer(path):
    import geopandas as gpd
    return gpd.read_file(path)

def read_with_deadline(path, seconds=300):
    with ProcessPoolExecutor(max_workers=1) as pool:
        future = pool.submit(read_layer, path)
        try:
            return future.result(timeout=seconds)
        except FutureTimeout:
            for proc in pool._processes.values():
                proc.terminate()
            raise TimeoutError(f"reading {path} exceeded {seconds}s")

A thread cannot be killed in Python, so a separate process is the only reliable way to enforce a deadline on blocking C code.

Example 4: Per-file retries inside a batch loop

In a batch job, a retry belongs around each file, and a final failure should skip that file rather than end the run.

summary = []
for path in sorted(src_dir.glob("*.gpkg")):
    try:
        gdf = with_retry(gpd.read_file, path, attempts=3, base_delay=0.5)
        result = process(gdf)
        write_atomic(result, out_dir / path.name)
        summary.append({"file": path.name, "status": "ok"})
    except Exception as exc:
        log.exception("permanently failed: %s", path.name)
        summary.append({"file": path.name, "status": f"failed: {exc}"})

The retry handles the blip; the try/except handles the genuinely bad file. See batch process a folder of GIS files.

Example 5: A circuit breaker for a service that is simply down

When a service is down, retrying every file wastes the whole run. Stop after enough consecutive failures.

class CircuitBreaker:
    def __init__(self, threshold=5):
        self.threshold, self.failures = threshold, 0

    def call(self, fn, *args, **kwargs):
        if self.failures >= self.threshold:
            raise RuntimeError(
                f"circuit open: {self.failures} consecutive failures — service is down")
        try:
            result = fn(*args, **kwargs)
        except Exception:
            self.failures += 1
            raise
        self.failures = 0
        return result

breaker = CircuitBreaker(threshold=5)
for bbox in tiles:
    gdf = breaker.call(with_retry, fetch_features, WFS_URL, bbox)

Explanation

Retries are a bet that the world will be different in a second. That bet pays off for anything crossing a network or a filesystem boundary, and never pays off for logic errors or bad data. Splitting failures into transient and permanent is therefore not a refinement — it is the design. A job that retries FileNotFoundError four times takes seven extra seconds to deliver exactly the same message, and it teaches whoever reads the log to distrust retry warnings entirely.

Exponential backoff exists because the most common cause of a transient failure is load, and immediate retries add load. Doubling the delay gives the other side room; jitter prevents synchronised herds of retries from re-creating the spike. Both matter more as soon as you run work in parallel, where a dozen workers can hammer a struggling server simultaneously.

Without a timeout a hung call runs indefinitely; with one it fails at a known deadline and can be retried. — A retry without a timeout is not a safety net — one hung call holds the job open forever.

Timeouts are the more important half of this article, and the one more often skipped. A retry policy without timeouts is not a safety net — a single hung connection can hold a job open indefinitely, past the next scheduled run, until someone notices the following week. The rule is absolute: every call that leaves the process gets a deadline. Where the library offers no timeout, put the work in a subprocess you can kill, as in Example 3.

Idempotence makes retries safe. Because a retried operation may run more than once, any side effect must tolerate repetition: write to a temporary file and rename, use a transaction and commit once, or derive the output name from the input so a repeat overwrites its own work rather than appending. Get this wrong and a retry produces a half-written GeoPackage or a table with duplicated rows — a worse outcome than the original failure. The same reasoning underpins scheduling, where a job may be run twice by a scheduler that catches up after downtime.

Finally, retries hide problems by design, so make sure they are visible. Log every retry at warning level with the reason, and count them. A job that quietly retries twice on every run is telling you something about the service it depends on, and that information is worth more than the seconds the retries saved.

Edge cases or notes

Retrying a partially written file

If the first attempt failed halfway through writing, the second may append to or trip over the debris. Always write to a temporary path and replace() it into place. Path.replace() is atomic within one filesystem, so a reader sees either the old file or the new one, never a half.

Retries inside parallel workers

Each worker retries independently, so a service already under strain receives workers × attempts requests. Jitter helps; a shared circuit breaker helps more. When throughput matters more than latency, reduce the worker count rather than the retry count.

`requests` does not raise on HTTP errors

requests.get() returns normally for a 500, so a naïve try/except around it retries nothing. Either check resp.status_code explicitly or call resp.raise_for_status() — and remember that turns 4xx into exceptions too, which you should not retry.

Timeouts through GDAL

gpd.read_file() on a URL delegates to GDAL's HTTP driver, which ignores Python timeout arguments. Configure GDAL_HTTP_TIMEOUT, GDAL_HTTP_CONNECTTIMEOUT, and optionally GDAL's own GDAL_HTTP_MAX_RETRY and GDAL_HTTP_RETRY_DELAY. Set them before the first GDAL call, and be aware GDAL's built-in retries stack with yours.

Retrying a database write

A retried INSERT can duplicate rows if the first attempt actually succeeded and only the acknowledgement was lost. Wrap writes in an explicit transaction, or make them idempotent with a unique key and an upsert. This is the one case where a retry can silently corrupt data rather than merely waste time.

Total time budget

Four attempts at 1, 2, 4, and 8 seconds is fifteen seconds — fine. The same policy applied per file across 500 files, with a service that is down, is over two hours of sleeping. Bound the job, not just each call: track elapsed time and abort the run when a budget is exhausted.

Internal links

FAQ

Which errors should I retry in a GIS job?

Only those a second attempt could plausibly fix: timeouts, connection resets, temporarily locked files, and HTTP 429/5xx responses. Never retry missing files, permission errors, invalid CRS codes, malformed geometry, or HTTP 4xx — they will fail identically every time, and retrying them delays the real error message.

How many retries and how long should I wait?

Three or four attempts with exponential backoff starting at one second covers nearly all transient failures — roughly seven to fifteen seconds total. Beyond that you are usually waiting on something genuinely down, where failing quickly and alerting is more useful than continuing to wait.

What is jitter and do I need it?

Jitter is a small random addition to each backoff delay. It matters when several workers fail at the same moment: without it they all retry at exactly the same instant and re-create the overload. With parallel processing or a shared service, add it; for a single-threaded job it is harmless insurance.

How do I put a timeout on `gpd.read_file()` over HTTP?

Through GDAL, not Python. Set GDAL_HTTP_TIMEOUT and GDAL_HTTP_CONNECTTIMEOUT in the environment before the first GDAL call. For anything else with no timeout support, run it in a ProcessPoolExecutor with future.result(timeout=…) and terminate the process if it overruns — a thread cannot be interrupted while blocked in C code.

Is it safe to retry a write?

Only if the write is idempotent. Write to a temporary file and replace() it into the final path so a repeat cannot leave partial output. For databases, use a transaction or an upsert with a unique key; a plain retried INSERT can duplicate rows when the first attempt succeeded but its acknowledgement was lost.

Should I retry the whole pipeline stage or just the failing call?

Just the call. Retrying a stage repeats work that already succeeded, can double side effects, and makes the log harder to read. Wrap the single external operation — one HTTP request, one file read, one query — and let everything around it stay simple.

How do I stop retries from hiding a service that is down?

Log every retry at warning level and count them, so a rising retry rate is visible before it becomes an outage. Add a circuit breaker that stops trying after several consecutive failures, and make the final failure message say how many attempts were made over what period. Then alert on it — a job that silently retried for an hour tells nobody anything.

How to Add Retries and Timeouts to an Automated GIS Job in Python #

Problem statement #

Quick answer #

Step-by-step solution #

Sort failures into transient and permanent #

Back off exponentially, and add jitter #

Always pass a timeout #

Wrap the call, not the pipeline #

Make retried operations safe to repeat #

Use a library when the rules get complicated #

Give up loudly #

Code examples #

Example 1: A reusable decorator #

Example 2: Retrying HTTP by status code #

Example 3: A hard timeout around code that does not accept one #

Example 4: Per-file retries inside a batch loop #

Example 5: A circuit breaker for a service that is simply down #

Explanation #

Edge cases or notes #

Retrying a partially written file #

Retries inside parallel workers #

requests does not raise on HTTP errors #

Timeouts through GDAL #

Retrying a database write #

Total time budget #

Internal links #

FAQ #

Which errors should I retry in a GIS job? #

How many retries and how long should I wait? #

What is jitter and do I need it? #

How do I put a timeout on gpd.read_file() over HTTP? #

Is it safe to retry a write? #

Should I retry the whole pipeline stage or just the failing call? #

How do I stop retries from hiding a service that is down? #