🔧 Fix Batch processing Performance & scale Batch Processing · step 26 of 26 Updated

Parallel GIS Batch Job Hangs or Crashes with multiprocessing: How to Fix It

Problem statement

The serial version of the batch works. You swap the loop for a ProcessPoolExecutor and one of three things happens: the script prints nothing and never exits, it raises a pickling error before any work starts, or — the most alarming one — it starts the whole script again, over and over.

_pickle.PicklingError: Can't pickle : it's not the same object
RuntimeError: An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
concurrent.futures.process.BrokenProcessPool: A process in the process pool
was terminated abruptly while the future was running or pending.

Common causes:

  • no if __name__ == "__main__": guard, so every child re-imports and re-runs the script
  • the worker function or its arguments cannot be pickled (lambdas, open datasets, layer objects)
  • a worker was killed by the OOM killer, which surfaces as BrokenProcessPool
  • a native GDAL/GEOS object was created before the fork and used after it
  • results are collected with pool.map on a generator that itself opens files
  • workers all write to the same output file and block on a lock
  • the pool is created inside a Jupyter notebook, where fork and the interactive session interact badly

Parallelism does not change what your per-file function does — it changes what has to cross a process boundary, and when. Nearly every hang or crash comes from that boundary.

Quick answer

To make a parallel GIS batch behave:

  1. put the launcher behind if __name__ == "__main__": — always, on every platform
  2. make the worker a top-level, module-level function that takes and returns paths and plain data
  3. open datasets inside the worker; never pass an open GeoDataFrame handle, connection, or QGIS object
  4. give each worker its own output file
  5. read results with as_completed() and wrap future.result() in try/except
from concurrent.futures import ProcessPoolExecutor, as_completed
from pathlib import Path
import geopandas as gpd

def process_one(path_str: str) -> tuple[str, int]:
    path = Path(path_str)                     # open inside the worker
    gdf = gpd.read_file(path)
    out = Path("data/out") / f"{path.stem}.gpkg"
    gdf.to_file(out, driver="GPKG")           # unique target per worker
    return path.name, len(gdf)

if __name__ == "__main__":                    # required
    files = [str(p) for p in sorted(Path("data/raw").glob("*.shp"))]
    Path("data/out").mkdir(parents=True, exist_ok=True)
    with ProcessPoolExecutor(max_workers=4) as pool:
        futures = {pool.submit(process_one, f): f for f in files}
        for fut in as_completed(futures):
            try:
                name, n = fut.result()
                print(f"ok {name}: {n} features")
            except Exception as exc:
                print(f"failed {futures[fut]}: {type(exc).__name__}: {exc}")

Everything the worker needs is a string; everything it returns is a string and an int. That is the shape that never gives trouble.

What crosses the process boundary

Parent process sending file paths to workers, each returning small result records.
Send paths, not data. Each worker opens, processes and writes on its own.

Step-by-step solution

Triage table of multiprocessing failure causes and their fixes.
Six failures, six fixes — and only one of them is about performance.

Add the __main__ guard

On Windows and macOS, new processes are created with spawn: the child starts a fresh interpreter and imports your script to find the worker function. Without a guard, that import runs your launcher again, which starts more children, which import again.

# module level: definitions only
def process_one(path_str: str):
    ...

# everything that *does* something goes here
if __name__ == "__main__":
    main()

The RuntimeError about "bootstrapping phase" is Python catching this for you. A silent hang or a fork bomb is what happens when it cannot.

Make the worker picklable

With spawn, the worker function is located by name and its arguments are pickled. That rules out several convenient things:

# not picklable — a lambda has no importable name
pool.map(lambda p: process(p, buffer=100), files)

# not picklable — a closure over local state
def make_worker(dist):
    def work(p): return process(p, dist)
    return work

# picklable — a module-level function plus a partial
from functools import partial
pool.map(partial(process_one, buffer=100), files)

The same applies to arguments. Paths, numbers, strings, dicts and DataFrames pickle fine. Open file handles, database connections, rasterio datasets, fiona collections and any PyQGIS object do not.

Do the opening inside the worker

Even when an object technically pickles, sending a large GeoDataFrame to every worker copies it into every process. Send the path and let the worker read it.

# bad: serialises the whole frame N times
pool.map(partial(clip_to, boundary=big_gdf), files)

# good: each worker reads the small boundary once, itself
BOUNDARY = "data/ref/boundary.gpkg"

def process_one(path_str):
    boundary = gpd.read_file(BOUNDARY)      # cheap, local to the worker
    gdf = gpd.read_file(path_str)
    return gpd.clip(gdf, boundary).shape[0]

If the shared input is expensive to load, use an initialiser that runs once per worker:

_boundary = None

def init_worker(path):
    global _boundary
    _boundary = gpd.read_file(path)

with ProcessPoolExecutor(max_workers=4, initializer=init_worker, initargs=(BOUNDARY,)) as pool:
    ...

Choose the start method deliberately

fork (the Linux default before 3.14) is fast but copies the parent's memory and threads, which is unsafe with libraries that hold native state — GDAL, GEOS, PROJ, PyQGIS and anything using OpenMP. spawn is slower to start and safest.

import multiprocessing as mp

if __name__ == "__main__":
    mp.set_start_method("spawn", force=True)     # before creating any pool
    main()

If a parallel job deadlocks immediately on Linux but works on Windows, an unsafe fork is the first thing to suspect.

Give every worker its own output

Two processes writing one GeoPackage will corrupt it, and two processes appending to one shapefile will interleave garbage. Write per-worker files and merge afterwards.

def process_one(path_str: str) -> str:
    path = Path(path_str)
    out = Path("data/out/parts") / f"{path.stem}.gpkg"
    gpd.read_file(path).to_file(out, driver="GPKG")
    return str(out)

# after the pool closes
import pandas as pd
parts = [gpd.read_file(p) for p in sorted(Path("data/out/parts").glob("*.gpkg"))]
merged = gpd.GeoDataFrame(pd.concat(parts, ignore_index=True), crs=parts[0].crs)
merged.to_file("data/out/merged.gpkg", driver="GPKG")

Stop workers fighting over threads

GDAL, NumPy and PROJ may each start their own thread pools. Four processes times eight threads is thirty-two threads on eight cores — all of them slower than one. Cap the native thread counts before importing the libraries.

import os
for var in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "GDAL_NUM_THREADS"):
    os.environ.setdefault(var, "1")

import geopandas as gpd     # imports must come after the env vars

A parallel job that is slower than the serial one is usually oversubscription, not overhead.

Handle BrokenProcessPool for what it is

This exception means a worker died without reporting — nearly always the OOM killer, sometimes a segfault inside a native library. It is not a bug in your Python code, and no try inside the worker can catch it.

from concurrent.futures.process import BrokenProcessPool

try:
    with ProcessPoolExecutor(max_workers=4) as pool:
        ...
except BrokenProcessPool:
    print("a worker died — reduce max_workers or the per-file memory", file=sys.stderr)

Halve max_workers, then check per-file peak memory: total usage is roughly workers × per-file peak.

Code examples

Example 1: the complete pattern 

from concurrent.futures import ProcessPoolExecutor, as_completed
from pathlib import Path
import multiprocessing as mp
import os, sys

for var in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "GDAL_NUM_THREADS"):
    os.environ.setdefault(var, "1")

import geopandas as gpd

SRC = Path("data/raw")
OUT = Path("data/out/parts")

def process_one(path_str: str) -> dict:
    path = Path(path_str)
    gdf = gpd.read_file(path, columns=["id", "class", "geometry"])
    gdf = gdf[gdf.geometry.notna()]
    dest = OUT / f"{path.stem}.gpkg"
    gdf.to_file(dest, driver="GPKG")
    return {"file": path.name, "features": len(gdf), "output": str(dest)}

def main() -> int:
    OUT.mkdir(parents=True, exist_ok=True)
    files = [str(p) for p in sorted(SRC.rglob("*.shp"))]
    if not files:
        print("nothing to do", file=sys.stderr)
        return 2

    ok, failed = [], []
    with ProcessPoolExecutor(max_workers=os.cpu_count() // 2 or 1) as pool:
        futures = {pool.submit(process_one, f): f for f in files}
        for i, fut in enumerate(as_completed(futures), start=1):
            src = futures[fut]
            try:
                ok.append(fut.result())
            except Exception as exc:
                failed.append((src, f"{type(exc).__name__}: {exc}"))
            print(f"[{i}/{len(files)}] {Path(src).name}", flush=True)

    print(f"\n{len(ok)} ok, {len(failed)} failed")
    for src, err in failed:
        print(f"  ! {src}: {err}")
    return 1 if failed else 0

if __name__ == "__main__":
    mp.set_start_method("spawn", force=True)
    raise SystemExit(main())

Example 2: a per-task timeout so one file cannot hang the run 

from concurrent.futures import ProcessPoolExecutor, as_completed, TimeoutError

with ProcessPoolExecutor(max_workers=4) as pool:
    futures = {pool.submit(process_one, f): f for f in files}
    for fut in as_completed(futures, timeout=None):
        try:
            print(fut.result(timeout=300))          # five minutes per file
        except TimeoutError:
            print(f"timed out: {futures[fut]}", file=sys.stderr)

Note that a timeout stops you waiting; it does not kill the worker. For a hard kill, run the step as a subprocess with subprocess.run(..., timeout=...).

Example 3: threads instead of processes for I/O-bound work

Reading many small files over a network share is I/O-bound. Threads avoid pickling entirely and share memory.

from concurrent.futures import ThreadPoolExecutor

def fetch_info(path_str: str) -> dict:
    import pyogrio
    info = pyogrio.read_info(path_str)
    return {"file": path_str, "features": info["features"], "crs": info["crs"]}

with ThreadPoolExecutor(max_workers=16) as pool:
    for rec in pool.map(fetch_info, files):
        print(rec)

GDAL releases the GIL during I/O, so threads genuinely overlap here. For CPU-bound geometry work, processes remain the right tool.

Example 4: recycle workers to cap a leak 

with ProcessPoolExecutor(max_workers=4, max_tasks_per_child=20) as pool:
    list(pool.map(process_one, files))

Each worker exits after twenty files and is replaced, which bounds any per-worker memory growth. On Python below 3.11, use multiprocessing.Pool(4, maxtasksperchild=20).

Explanation

A process pool is not a faster loop. It is several independent Python interpreters, and everything they share must be serialised, sent, and rebuilt on the other side. That single fact explains the pickling errors, the re-import behaviour, and most of the hangs.

Comparison grid of fork and spawn start methods across safety, speed and platform.
`fork` copies the parent, `spawn` rebuilds from imports — the difference decides what is safe.

With spawn, the child interpreter has none of the parent's state. To call process_one, it imports the module that defines it — your script — and looks the name up. Any top-level code runs during that import, which is why the __main__ guard is not a style preference: without it, the import is your program, and it starts recursively.

With fork, the child is a copy of the parent's memory, so imports and objects come for free. The catch is that only the calling thread survives, and native libraries can hold locks or handles that were owned by threads that no longer exist. GDAL, GEOS and PROJ all keep such state, which is why a forked worker can deadlock on its first read with no error message at all.

Deciding what to send follows from this. A path is a few bytes and re-opening a file in the worker is cheap; a GeoDataFrame is megabytes that must be pickled, copied through a pipe, and unpickled — often costing more than the work itself. The same logic applies to returns: send back counts, paths and status strings, not frames.

Finally, resources that are not memory do not parallelise just because processes do. One GeoPackage cannot take concurrent writers. One disk has a fixed number of I/O operations per second. One machine has fixed RAM, and four workers multiply the per-file peak by four. Most "parallelism made it slower" reports are one of those three limits, not Python overhead.

Edge cases or notes

  • Jupyter and spawn do not mix well: The notebook module cannot be re-imported cleanly. Put the worker in a .py file and import it, or run the batch as a script.
  • fork is no longer the default everywhere: Python 3.14 changed the Linux default to forkserver. Set the method explicitly if your code depends on one.
  • PyQGIS objects never cross a process boundary: Initialise QgsApplication inside each worker, or run each QGIS task as a separate subprocess.
  • pool.map returns results in order but blocks: Use as_completed when you want progress reporting as files finish rather than in submission order.
  • A worker's print may be interleaved: Two processes writing the same terminal produce mixed lines. Return status records and log from the parent, or use a QueueHandler.
  • Keyboard interrupt handling is messy: Ctrl-C may leave orphaned children. with ProcessPoolExecutor(...) shuts down cleanly on exit; avoid creating pools without the context manager.

FAQ

Why does my script start itself over and over?

The launcher is not behind if __name__ == "__main__":. With the spawn start method each child imports your module to find the worker function, and any top-level code runs again — including the code that creates the pool.

What does BrokenProcessPool actually mean?

A worker process died without returning a result, usually killed by the OS for using too much memory, or crashed inside a native library. Reduce max_workers, then reduce per-file memory. No try inside the worker can catch it.

Why can't I pass an open GeoDataFrame or dataset to a worker?

Arguments are pickled to cross the process boundary. Open handles reference OS-level state that cannot be serialised, and large frames are expensive to copy. Pass the path and open it inside the worker.

Should I use threads or processes for GIS work?

Threads for I/O-bound work such as reading many small files or hitting an API — GDAL releases the GIL during I/O. Processes for CPU-bound geometry work such as buffers, overlays and dissolves.

Why is the parallel version slower than the serial one?

Usually thread oversubscription (each process spawning its own native thread pool), disk contention, or per-task overhead that exceeds the work. Set OMP_NUM_THREADS=1 and friends, and batch small files into larger tasks.

How do I get a progress bar with a process pool?

Submit futures into a dict and iterate as_completed(), incrementing a counter — or wrap that iterator in tqdm. pool.map yields in submission order, so it reports progress unevenly.

Can two workers write to the same GeoPackage?

No. It is SQLite underneath and concurrent writers from separate processes risk corruption. Write one file per worker into a parts/ folder and merge once the pool has closed.