⚙️ How-To

How to Speed Up Batch GIS Jobs with Parallel Processing in Python

Your batch script works, but it crawls. It reads a file, reprojects it, buffers some geometry, writes it out, then moves to the next one — while seven of your eight CPU cores sit idle. A folder of 500 files that each take four seconds is over half an hour of watching a progress line tick. The files are independent, the work is CPU-bound, and the machine has cores to spare. This guide shows you how to turn that serial loop into a parallel one with concurrent.futures.ProcessPoolExecutor, so the whole folder finishes in a fraction of the time — and when parallelism will not help, so you do not add complexity for nothing.

Problem statement

You have a working per-file batch loop, but throughput is the bottleneck. You are hitting some mix of these:

  • One core doing all the work — geometry operations (buffer, overlay, reproject, dissolve) are CPU-bound, and a plain for loop pins a single core while the rest of the machine idles.
  • Long wall-clock times — hundreds or thousands of independent files, each taking seconds, add up to jobs that run for tens of minutes or hours end to end.
  • Threads that did nothing — you tried threading and saw no speedup, because Python's GIL serialises the CPU-bound work anyway.
  • PicklingError when you reach for a pool — you wrapped the loop body in a lambda or a nested function and the executor refused to run it.
  • RuntimeError about a "current process" — you launched a process pool without an if __name__ == "__main__": guard and the program tried to fork itself endlessly.

The goal: one script that spreads independent per-file work across your CPU cores with processes, collects results and errors cleanly, and degrades gracefully back to serial when parallelism is the wrong tool.

Quick answer

Move the read-process-write body of your loop into a top-level worker function that takes a file path and returns a small result. Then hand every path to a ProcessPoolExecutor, which runs them across separate Python processes — sidestepping the GIL that would otherwise serialise CPU-bound geometry work. Guard the driver with if __name__ == "__main__":, collect results as they finish, and record any per-file exceptions instead of letting one bad file abort the run.

A queue of files distributed across several worker processes running concurrently, then results collected.
Each file is an independent task; a pool of worker processes runs them concurrently, one per CPU core, and the driver collects results.
import geopandas as gpd
from pathlib import Path
from concurrent.futures import ProcessPoolExecutor, as_completed

SRC = Path("data/raw")
OUT = Path("data/clean")
OUT.mkdir(parents=True, exist_ok=True)

def process_file(path_str):
    """Top-level worker: read one file, process it, write it. Returns a status tuple."""
    path = Path(path_str)
    gdf = gpd.read_file(path)
    gdf = gdf.to_crs("EPSG:3857")
    gdf["geometry"] = gdf.geometry.buffer(50)     # CPU-bound geometry work
    out_path = OUT / f"{path.stem}.gpkg"
    gdf.to_file(out_path, driver="GPKG")
    return (path.name, len(gdf), "ok")

if __name__ == "__main__":
    paths = [str(p) for p in sorted(SRC.glob("*.shp"))]
    results = []
    with ProcessPoolExecutor() as executor:
        futures = {executor.submit(process_file, p): p for p in paths}
        for fut in as_completed(futures):
            src = futures[fut]
            try:
                results.append(fut.result())
            except Exception as exc:
                results.append((Path(src).name, 0, f"error: {exc}"))

    for name, n, status in results:
        print(f"{name:30} {n:7} rows  | {status}")

That is the whole job. The rest of this article explains why the worker must live at module top level, when to prefer map over submit, how to size the pool, and when parallelism is the wrong answer entirely.

Step-by-step solution

Start from a working serial loop

Parallelism does not fix a broken loop; it multiplies whatever the loop already does, correct or not. Before you parallelise anything, get a plain serial version working end to end on a handful of files. The serial shapefile batch pattern is exactly this: find the files, run one function per file, write output, catch errors. Confirm the output is right first — a parallel run that produces garbage 8× faster is not progress.

import geopandas as gpd
from pathlib import Path

SRC = Path("data/raw")
OUT = Path("data/clean")
OUT.mkdir(parents=True, exist_ok=True)

for path in sorted(SRC.glob("*.shp")):
    gdf = gpd.read_file(path)
    gdf = gdf.to_crs("EPSG:3857")
    gdf["geometry"] = gdf.geometry.buffer(50)
    gdf.to_file(OUT / f"{path.stem}.gpkg", driver="GPKG")
    print(f"done {path.name}")

Extract the loop body into a top-level worker function

The single most important step: lift the read-process-write body into a function defined at module top level — not nested inside another function, not a lambda. Give it one simple argument (a path string) and have it return a small, simple result. A process pool has to send this function to each worker process by pickling a reference to it, and only functions importable by name from the top of a module can be pickled. This is the rule that trips up almost everyone the first time.

import geopandas as gpd
from pathlib import Path

OUT = Path("data/clean")

def process_file(path_str):
    path = Path(path_str)
    gdf = gpd.read_file(path)
    gdf = gdf.to_crs("EPSG:3857")
    gdf["geometry"] = gdf.geometry.buffer(50)
    gdf.to_file(OUT / f"{path.stem}.gpkg", driver="GPKG")
    return (path.name, len(gdf), "ok")

Pass a str rather than a Path object if you want to be maximally conservative about pickling, though modern pathlib.Path pickles fine. Never pass the whole GeoDataFrame into the pool — serialising large geometries between processes is expensive. Send the path and let each worker read its own file.

Fan the files out across a ProcessPoolExecutor

With a picklable worker in hand, hand every path to a ProcessPoolExecutor. The with block spins up a pool of worker processes, dispatches the tasks, and shuts the pool down cleanly on exit. The simplest dispatch is executor.map, which applies the worker to every path and yields results in input order.

from concurrent.futures import ProcessPoolExecutor
from pathlib import Path

if __name__ == "__main__":
    paths = [str(p) for p in sorted(Path("data/raw").glob("*.shp"))]

    with ProcessPoolExecutor() as executor:
        for result in executor.map(process_file, paths):
            print(result)

executor.map is the closest analogue to a plain for loop and is the right default when you want results in order and are happy for the first exception to propagate and stop the run.

Guard the entry point with if __name__ == "__main__":

This is not optional on Windows or macOS (which use the spawn start method), and it is good hygiene everywhere. To create a worker process, Python re-imports your module in the child. Without the guard, the module-level code that starts the pool runs again in every child, which tries to start more pools, and you get a RuntimeError or a fork bomb of processes. Put every line that launches work inside the guard; keep function and constant definitions at top level so the children can import them.

# top level: definitions the workers need (imported by every child)
def process_file(path_str):
    ...

# guarded: the driver that actually launches the pool (runs once, in the parent)
if __name__ == "__main__":
    with ProcessPoolExecutor() as executor:
        ...

Collect results and errors as files finish

For real batches you want two things map does not give you easily: per-file error isolation, and progress feedback as work completes rather than in submission order. Use executor.submit to schedule each task, keep a mapping from future back to its source path, and drain them with as_completed. Calling future.result() inside a try/except re-raises whatever the worker raised — so one corrupt file becomes a logged row, not a dead batch.

from concurrent.futures import ProcessPoolExecutor, as_completed
from pathlib import Path

if __name__ == "__main__":
    paths = [str(p) for p in sorted(Path("data/raw").glob("*.shp"))]
    results, done = [], 0

    with ProcessPoolExecutor() as executor:
        futures = {executor.submit(process_file, p): p for p in paths}
        for fut in as_completed(futures):
            src = Path(futures[fut]).name
            done += 1
            try:
                results.append(fut.result())
                print(f"[{done}/{len(paths)}] ok   {src}")
            except Exception as exc:
                results.append((src, 0, f"error: {exc}"))
                print(f"[{done}/{len(paths)}] FAIL {src}: {exc}")

Tune the number of workers

By default ProcessPoolExecutor uses roughly os.cpu_count() workers, which is a sensible start for CPU-bound work. Set max_workers explicitly when you need to leave headroom for other processes, or when each worker's memory footprint means you cannot afford one per core. More workers than cores rarely helps CPU-bound geometry work and just adds context-switching and memory pressure.

import os
from concurrent.futures import ProcessPoolExecutor

workers = max(1, (os.cpu_count() or 2) - 1)   # leave one core for the OS / you

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=workers) as executor:
        ...

Code examples

Example 1: The baseline serial loop (for contrast)

Time this first so you have a number to beat. Everything below is measured against it.

import time
import geopandas as gpd
from pathlib import Path

SRC = Path("data/raw")
OUT = Path("data/clean")
OUT.mkdir(parents=True, exist_ok=True)

start = time.perf_counter()
for path in sorted(SRC.glob("*.shp")):
    gdf = gpd.read_file(path)
    gdf = gdf.to_crs("EPSG:3857")
    gdf["geometry"] = gdf.geometry.buffer(50)
    gdf.to_file(OUT / f"{path.stem}.gpkg", driver="GPKG")

print(f"serial: {time.perf_counter() - start:.1f} s")

Example 2: ProcessPoolExecutor with executor.map

The minimal parallel version. Results come back in input order, and the first unhandled exception stops the run — fine when you just want raw speed on clean data.

import time
import geopandas as gpd
from pathlib import Path
from concurrent.futures import ProcessPoolExecutor

SRC = Path("data/raw")
OUT = Path("data/clean")
OUT.mkdir(parents=True, exist_ok=True)

def process_file(path_str):
    path = Path(path_str)
    gdf = gpd.read_file(path)
    gdf = gdf.to_crs("EPSG:3857")
    gdf["geometry"] = gdf.geometry.buffer(50)
    gdf.to_file(OUT / f"{path.stem}.gpkg", driver="GPKG")
    return path.name

if __name__ == "__main__":
    paths = [str(p) for p in sorted(SRC.glob("*.shp"))]
    start = time.perf_counter()
    with ProcessPoolExecutor() as executor:
        for name in executor.map(process_file, paths):
            print(f"done {name}")
    print(f"parallel: {time.perf_counter() - start:.1f} s")

Example 3: submit + as_completed with error isolation and a progress count

The version you actually want for production batches: per-file exceptions are caught and logged, and progress prints as each file finishes rather than in order.

import geopandas as gpd
from pathlib import Path
from concurrent.futures import ProcessPoolExecutor, as_completed

SRC = Path("data/raw")
OUT = Path("data/clean")
OUT.mkdir(parents=True, exist_ok=True)

def process_file(path_str):
    path = Path(path_str)
    gdf = gpd.read_file(path)
    if gdf.crs is None:
        raise ValueError("missing CRS")
    gdf = gdf.to_crs("EPSG:3857")
    gdf["geometry"] = gdf.geometry.buffer(50)
    gdf.to_file(OUT / f"{path.stem}.gpkg", driver="GPKG")
    return (path.name, len(gdf))

if __name__ == "__main__":
    paths = [str(p) for p in sorted(SRC.glob("*.shp"))]
    ok, failed, done = [], [], 0

    with ProcessPoolExecutor() as executor:
        futures = {executor.submit(process_file, p): p for p in paths}
        for fut in as_completed(futures):
            src = Path(futures[fut]).name
            done += 1
            try:
                name, n = fut.result()
                ok.append(name)
                print(f"[{done}/{len(paths)}] ok   {name} ({n} rows)")
            except Exception as exc:
                failed.append((src, str(exc)))
                print(f"[{done}/{len(paths)}] FAIL {src}: {exc}")

    print(f"\ndone: {len(ok)} ok, {len(failed)} failed")
    for name, err in failed:
        print(f"- {name}: {err}")

Example 4: Choosing max_workers with os.cpu_count()

Size the pool to the machine and the memory each worker needs. os.cpu_count() can return None in odd environments, so guard it, and floor the result at 1.

import os
from concurrent.futures import ProcessPoolExecutor

cores = os.cpu_count() or 2

# CPU-bound geometry work: about one worker per core, minus one for headroom.
cpu_workers = max(1, cores - 1)

# Memory-heavy files: cap workers so peak RSS stays under your RAM budget.
gb_per_worker = 2
ram_budget_gb = 12
mem_workers = max(1, ram_budget_gb // gb_per_worker)

workers = min(cpu_workers, mem_workers)
print(f"{cores} cores detected -> using {workers} workers")

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=workers) as executor:
        ...   # dispatch as in Example 3

Example 5: ThreadPoolExecutor for I/O-bound steps (downloads)

When the bottleneck is waiting on the network or disk rather than the CPU — downloading tiles, fetching remote files, calling an API — threads are the right tool. The GIL is released during I/O waits, so threads overlap the waiting without the memory cost of processes. Note the near-identical API: swap ProcessPoolExecutor for ThreadPoolExecutor.

import urllib.request
from pathlib import Path
from concurrent.futures import ThreadPoolExecutor, as_completed

DEST = Path("data/downloads")
DEST.mkdir(parents=True, exist_ok=True)

def download(url):
    name = url.rsplit("/", 1)[-1]
    urllib.request.urlretrieve(url, DEST / name)   # I/O-bound wait
    return name

if __name__ == "__main__":
    urls = [
        "https://example.com/tiles/a.gpkg",
        "https://example.com/tiles/b.gpkg",
        "https://example.com/tiles/c.gpkg",
    ]
    # Threads can far exceed core count for I/O-bound work.
    with ThreadPoolExecutor(max_workers=8) as executor:
        futures = {executor.submit(download, u): u for u in urls}
        for fut in as_completed(futures):
            try:
                print(f"downloaded {fut.result()}")
            except Exception as exc:
                print(f"failed {futures[fut]}: {exc}")

Explanation

The GIL, and why CPU-bound work needs processes

CPython has a Global Interpreter Lock (GIL): only one thread can execute Python bytecode at a time within a single process. For CPU-bound work — and buffering, overlaying, reprojecting, and dissolving geometry is CPU-bound — threads take turns on one core and give you no real speedup. Processes each have their own interpreter and their own GIL, so they genuinely run in parallel across cores. That is why ProcessPoolExecutor, not ThreadPoolExecutor, is the tool for geometry batches. (Some of GeoPandas' heavy lifting drops into GDAL/GEOS C code that can release the GIL, but you cannot rely on that for a whole pipeline — processes are the robust default.)

Processes versus threads at a glance

  • Processes (ProcessPoolExecutor): true parallelism for CPU-bound work; each worker is a separate Python interpreter with its own memory. The cost is startup time and memory per worker, plus the need to pickle arguments and results across the process boundary.
  • Threads (ThreadPoolExecutor): cheap and share memory, but the GIL serialises CPU-bound Python. They shine for I/O-bound work — network, disk, database — where each thread spends most of its time waiting, and the GIL is released during that wait.

The rule of thumb: CPU-bound → processes, I/O-bound → threads. A batch that downloads files and then crunches their geometry is both, and can use a thread pool for the fetch stage and a process pool for the compute stage.

Why lambdas and closures cannot be workers

To run your function in another process, the pool pickles a reference to it and the child re-imports it by qualified name. A lambda, a function defined inside another function, or a method bound to a local object has no importable top-level name, so pickling fails with PicklingError or AttributeError: Can't pickle local object. The fix is always the same: define the worker at module top level. If it needs extra configuration, pass that in as arguments, or bake fixed parameters in with functools.partial (which is picklable when it wraps a top-level function). The same constraint applies to arguments and return values — keep them small and picklable (paths, numbers, short tuples), never open file handles or database connections.

Edge cases or notes

When parallelism does not help

Processes are not free. If your files are tiny or the per-file work is trivial, the overhead of spawning workers, pickling arguments, and shuttling results can cost more than you save — a parallel run can be slower than serial. Parallelism pays off when the per-file work is substantial and CPU-bound and you have many independent files. For a dozen quick files, keep the plain loop.

I/O-bound batches see little gain from processes

If your job spends its time reading and writing rather than computing — many small files, a slow disk, or network storage — adding CPU worker processes will not speed it up, because the disk is the bottleneck, not the cores. Profile first. For genuinely I/O-bound stages use threads (Example 5), and consider faster formats and a faster disk before reaching for more workers.

Parallelism multiplies memory

Each worker process loads its own file and holds its own GeoDataFrame, so peak memory is roughly max_workers × the size of one working set. Eight workers on files that each need 2 GB is 16 GB before you count overhead. If you are already close to the edge on a single file, running eight at once will push you into an out-of-memory kill. Cap max_workers to fit your RAM budget (Example 4), and see fixing memory errors in GeoPandas when individual files are the problem.

Writing to the same output file from many workers

Workers run concurrently, so never have two of them write to the same output file or the same layer of one GeoPackage — you will get corruption or lock errors. Give each worker its own output path (one file per input, as in the examples). If you must combine everything into one dataset, write separate files in parallel and merge them in a single serial step afterwards.

One bad file should never kill the batch

With executor.map, the first unhandled exception propagates and stops iteration. For long overnight runs use submit + as_completed and wrap each future.result() in try/except (Example 3), so a corrupt geometry or unreadable file is logged and skipped while every other file still processes. To make the whole job restartable after a crash, combine this with a resumable batch pattern that skips already-written outputs.

Debugging is harder across processes

Tracebacks from a worker are re-raised in the parent when you call result(), but stepping through with a debugger, sharing global state, and printing in real time all behave differently across processes. Develop and debug the worker function serially — call process_file(one_path) directly, outside any pool — until it is correct, and only then wrap it in the executor. For durable diagnostics on a big run, add structured logging and an error report rather than relying on print.

FAQ

Should I use processes or threads to speed up my GIS batch?

Use processes (ProcessPoolExecutor) when the work is CPU-bound — buffering, overlay, reprojection, dissolve — because the GIL prevents threads from running Python bytecode in parallel. Use threads (ThreadPoolExecutor) when the work is I/O-bound, such as downloading files or waiting on a database, where each worker spends its time waiting and the GIL is released. Most geometry batches are CPU-bound, so processes are the default.

Why do I get a PicklingError or "Can't pickle local object"?

Your worker function (or one of its arguments) cannot be pickled, which the pool needs in order to send it to another process. This happens when the worker is a lambda, a nested/closure function, or a bound method, or when you pass something unpicklable like an open file handle. Define the worker at module top level, and pass only simple values (path strings, numbers) as arguments.

Why do I need if __name__ == "__main__":?

Creating a worker process re-imports your module in the child. Without the guard, the code that launches the pool runs again in every child, spawning more pools — a RuntimeError or a runaway fork of processes. Put function and constant definitions at top level so children can import them, and put everything that starts work inside the guard so it runs only in the parent.

How many workers should I use?

Start near os.cpu_count() for CPU-bound work, and subtract one to leave the OS a core. Reduce it further if each worker uses a lot of memory: max_workers should be low enough that max_workers × per-worker RAM fits comfortably in your machine. More workers than cores rarely helps CPU-bound work and just adds overhead. For I/O-bound thread pools you can safely use far more workers than cores.

When should I use executor.map versus submit + as_completed?

Use executor.map for the simple case: results in input order, and you are happy for the first exception to stop the run. Use submit + as_completed when you want per-file error isolation (wrap each result() in try/except), progress feedback as files finish, or to associate each result with its source path. Production batches almost always want the submit version.

My parallel version is slower than the serial one — why?

Either the files are too small or the per-file work too light for parallelism to pay off (process startup and pickling overhead dominates), or the job is I/O-bound and the disk, not the CPU, is the bottleneck. Adding CPU worker processes cannot speed up a disk-limited job. Profile the serial run first; parallelise only when the work is substantial, CPU-bound, and spread over many independent files.

Will running eight workers make me run out of memory?

It can. Each process holds its own GeoDataFrame, so peak memory is roughly max_workers × one file's working set. If a single file already uses a large fraction of your RAM, running several at once will trigger an out-of-memory kill. Cap max_workers to fit your RAM budget, shrink each file's footprint first (see the memory-errors guide), or process the largest files serially.

Can I share a progress bar or counter across workers?

Not directly through a shared variable — each process has its own memory, so a plain counter in the parent will not see the children's increments. The clean pattern is to increment a counter in the parent as each future completes in the as_completed loop (as in Example 3), which is where all results funnel back regardless of which worker produced them. That gives you an accurate running count without any cross-process shared state.