Threads, Processes and the GIL: Parallelism in Python GIS Explained
On this page
Problem statement
Eight threads, and the job gets slower. This is a per-feature Python calculation — each parcel's area through apply — on HM Land Registry's 338,611 Birmingham parcels (set up under Code examples), serially and split across eight threads:
import time
from concurrent.futures import ThreadPoolExecutor
def area_by_loop(chunk):
return chunk.apply(lambda g: g.area) # Python, one feature at a time
start = time.perf_counter()
serial = [area_by_loop(c) for c in chunks]
print(f"serial loop: {time.perf_counter() - start:.2f} s")
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=8) as pool:
threaded = list(pool.map(area_by_loop, chunks))
print(f"8 threads: {time.perf_counter() - start:.2f} s")serial loop: 0.49 s
8 threads: 1.14 s
The same eight threads on a geometry operation — a 25 m buffer of every parcel — do the opposite:
def buffer_chunk(chunk):
return chunk.buffer(25) # GEOS, which releases the GIL
start = time.perf_counter()
[buffer_chunk(c) for c in chunks]
print(f"serial buffer: {time.perf_counter() - start:.2f} s")
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=8) as pool:
list(pool.map(buffer_chunk, chunks))
print(f"8 threads: {time.perf_counter() - start:.2f} s")serial buffer: 6.63 s
8 threads: 1.31 s
Both outcomes are predictable once you know two things: what Python's Global Interpreter Lock actually locks, and where your GIS work is really spending its time. Geospatial Python is unusual here — much of the heavy lifting happens in C libraries that release the lock, so the usual "threads are useless in Python" advice is wrong about half the time.
Quick answer
Choose by what the work is waiting for:
- I/O-bound (reading files, network, database) → threads; GDAL releases the GIL during I/O
- CPU-bound in C (GEOS geometry ops, NumPy, rasterio) → threads often work, because those libraries release the GIL
- CPU-bound in Python (per-feature loops,
apply, dict building) → processes, or rewrite it vectorised - Whole-file batches → processes, one file per worker, no shared state
- Always measure; parallel overhead can exceed the saving
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import geopandas as gpd
# I/O-bound: many small reads → threads
def read_info(path):
import pyogrio
return pyogrio.read_info(path)
with ThreadPoolExecutor(max_workers=16) as pool:
infos = list(pool.map(read_info, paths))
# whole-file CPU work → processes (module-level function, paths not objects)
def process_one(path_str: str) -> str:
gdf = gpd.read_file(path_str)
out = gdf.to_crs(gdf.estimate_utm_crs()).buffer(25)
dest = path_str.replace("raw", "out")
gpd.GeoDataFrame(geometry=out).to_file(dest, driver="GPKG")
return dest
if __name__ == "__main__":
with ProcessPoolExecutor(max_workers=4) as pool:
list(pool.map(process_one, [str(p) for p in paths]))The if __name__ == "__main__" guard is not optional for processes, and passing paths rather than GeoDataFrames is what keeps the transfer cost near zero.
What the GIL actually blocks
The GIL, threads, processes and their costs
What the GIL is
The Global Interpreter Lock is a mutex inside CPython that ensures only one thread executes Python bytecode at a time. It exists because CPython's memory management — reference counting on every object — is not thread-safe, and a lock around the interpreter was far simpler and faster for single-threaded code than fine-grained locking everywhere.
The crucial detail is what it does not cover. A C extension can release the GIL while it works and reacquire it before touching Python objects. During that window other threads run freely.
# rough shape of a GIL-releasing C function
# Py_BEGIN_ALLOW_THREADS
# GEOSIntersection_r(handle, a, b); /* no Python objects touched */
# Py_END_ALLOW_THREADSShapely 2 releases the GIL around GEOS calls. GDAL releases it during I/O. NumPy releases it for most array operations. That is why threads are genuinely useful in a geospatial workload in a way they are not in, say, a pure-Python parser.
Find out what your work is actually doing
import time
import geopandas as gpd
def profile_step(fn, *args, label=""):
t0 = time.perf_counter()
cpu0 = time.process_time()
result = fn(*args)
wall = time.perf_counter() - t0
cpu = time.process_time() - cpu0
kind = "CPU-bound" if cpu / wall > 0.8 else "I/O-bound or waiting"
print(f"{label:28} wall {wall:6.2f}s cpu {cpu:6.2f}s → {kind}")
return result
gdf = profile_step(gpd.read_file, "data/parcels.gpkg", label="read_file")
profile_step(lambda g: g.to_crs(3857), gdf, label="to_crs")
profile_step(lambda g: g.buffer(25), gdf.geometry, label="buffer (GEOS)")
profile_step(lambda g: g.apply(lambda x: x.area), gdf.geometry, label="apply (Python loop)")read_file wall 0.50s cpu 0.50s → CPU-bound
to_crs wall 2.26s cpu 2.26s → CPU-bound
buffer (GEOS) wall 6.39s cpu 6.38s → CPU-bound
apply (Python loop) wall 0.45s cpu 0.45s → CPU-bound
If CPU time is close to wall time, the work is computing; if it is much lower, it is waiting. That single ratio tells you which parallelism to reach for — with one caveat the table shows: a read from a local disk the operating system has cached is computing, not waiting, and counts as CPU-bound.
Threads for I/O
from concurrent.futures import ThreadPoolExecutor, as_completed
import pyogrio
def inventory(paths, workers=16):
rows = []
with ThreadPoolExecutor(max_workers=workers) as pool:
futures = {pool.submit(pyogrio.read_info, p): p for p in paths}
for future in as_completed(futures):
path = futures[future]
try:
info = future.result()
rows.append({"path": str(path), "features": info["features"],
"crs": info["crs"]})
except Exception as exc:
rows.append({"path": str(path), "error": str(exc)})
return rows
import rasterio
def raster_header(path):
with rasterio.open(path) as src: # an HTTP range request for the header
return {"path": path, "size": (src.width, src.height), "crs": str(src.crs)}
batches = [remote_rasters[i * 24:(i + 1) * 24] for i in range(4)] # different files each run: no cache hits
runs = [({}, 1), ({}, 16), ({"GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR"}, 1),
({"GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR"}, 16)]
for (options, workers), batch in zip(runs, batches):
with rasterio.Env(**options):
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=workers) as pool:
headers = list(pool.map(raster_header, batch))
label = "EMPTY_DIR" if options else "default"
print(f"{label:9} {workers:>2} thread(s): {len(headers)} remote headers in {time.perf_counter() - start:5.1f} s")default 1 thread(s): 24 remote headers in 49.3 s
default 16 thread(s): 24 remote headers in 41.7 s
EMPTY_DIR 1 thread(s): 24 remote headers in 14.0 s
EMPTY_DIR 16 thread(s): 24 remote headers in 1.6 s
Reading headers over the network is almost entirely waiting — here, 24 Sentinel-2 band files on AWS per run, and a different 24 each time so none is served from GDAL's cache. With GDAL's defaults, though, sixteen threads barely help: on each open GDAL first tries to list the remote "directory", and those listing requests dominate and do not overlap. Tell it not to with GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR and each open is a couple of range requests — faster on its own, and sixteen threads then overlap the waits as they should. There is no pickling, no memory multiplication and no __main__ guard needed.
Threads for GEOS-bound geometry work
import numpy as np
def buffer_chunk(geoms):
return geoms.buffer(25) # GEOS releases the GIL
edges = np.linspace(0, len(gdf), 9).astype(int)
chunks = [gdf.geometry.iloc[a:b] for a, b in zip(edges[:-1], edges[1:])]
print(type(np.array_split(gdf.geometry, 8)[0]).__name__, "— what np.array_split returns now")
with ThreadPoolExecutor(max_workers=8) as pool:
parts = list(pool.map(buffer_chunk, chunks))
buffered = gpd.GeoSeries(np.concatenate([p.values for p in parts]), crs=gdf.crs)
print(len(buffered), buffered.crs)ndarray — what np.array_split returns now
338611 EPSG:27700
Slice with iloc, not np.array_split: with current NumPy and pandas, array_split on a GeoSeries hands back plain NumPy arrays of geometries, and .buffer fails on them.
This genuinely uses multiple cores, because each thread spends its time inside GEOS with the lock released. It also shares memory, so there is no per-worker copy of the data — the big advantage over processes.
Processes for Python-bound work
Whole-file batches go to processes, and the worker has to live in a real module so the child processes can import it. The folder here is the Birmingham parcels split into one file per ward, 69 files:
%%writefile ward_batch.py
from concurrent.futures import ProcessPoolExecutor
from pathlib import Path
import sys, time
import geopandas as gpd
def process_file(path_str: str) -> dict:
path = Path(path_str)
gdf = gpd.read_file(path)
gdf["footprint_m2"] = gdf.geometry.apply(lambda g: g.buffer(25).area) # Python per feature
dest = Path("data/out") / f"{path.stem}.gpkg"
gdf.to_file(dest, driver="GPKG")
return {"file": path.name, "features": len(gdf)}
if __name__ == "__main__":
Path("data/out").mkdir(parents=True, exist_ok=True)
paths = [str(p) for p in sorted(Path("data/wards").glob("*.gpkg"))]
workers = int(sys.argv[1])
start = time.perf_counter()
if workers == 0:
results = [process_file(p) for p in paths]
else:
with ProcessPoolExecutor(max_workers=workers) as pool:
results = list(pool.map(process_file, paths))
print(f"workers={workers}: {len(results)} files, {sum(r['features'] for r in results):,} parcels, "
f"{time.perf_counter() - start:.1f} s")import subprocess, sys
for workers in (0, 4, 8):
print(subprocess.run([sys.executable, "ward_batch.py", str(workers)], capture_output=True, text=True).stdout.strip())workers=0: 69 files, 337,909 parcels, 16.5 s
workers=4: 69 files, 337,909 parcels, 5.1 s
workers=8: 69 files, 337,909 parcels, 3.1 s
Whole-file batches are the ideal process workload: each worker is independent, arguments are small strings, and results are small dicts. The costs — start-up, pickling, memory per worker — are amortised over the work per file, and the per-feature Python loop that would hold the GIL in a thread runs in parallel because each process has a GIL of its own.
The trap: oversubscription
import os
# GDAL, NumPy's BLAS and PROJ each start their own thread pools
for var in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS",
"GDAL_NUM_THREADS", "NUMEXPR_NUM_THREADS"):
os.environ.setdefault(var, "1")
import geopandas as gpd # import after setting the variablesFour worker processes, each spawning eight BLAS threads, is thirty-two threads competing for eight cores. They spend their time context-switching, and the parallel version is slower than the serial one. Setting the native thread counts to 1 in worker processes is the standard fix — and the most common explanation for "parallel made it worse".
The other trap: transferring data
%%writefile transfer.py
from concurrent.futures import ProcessPoolExecutor
import resource, sys, time
import geopandas as gpd
def peak_mb():
return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
def count_rows(frame):
return len(frame), peak_mb()
def work_on_window(args):
path, start, count = args
part = gpd.read_file(path, engine="pyogrio", skip_features=start, max_features=count)
return len(part), peak_mb()
if __name__ == "__main__":
path = "data/parcels.gpkg"
start = time.perf_counter()
if sys.argv[1] == "send":
big = gpd.read_file(path) # the whole frame, pickled to every worker
with ProcessPoolExecutor(4) as pool:
results = list(pool.map(count_rows, [big] * 4))
else:
windows = [(path, s, 100_000) for s in range(0, 400_000, 100_000)]
with ProcessPoolExecutor(4) as pool: # each worker reads its own slice
results = list(pool.map(work_on_window, windows))
total = sum(rows for rows, _ in results)
print(f"{sys.argv[1]:7} {total:>9,} rows {time.perf_counter() - start:5.1f} s "
f"largest worker {max(mb for _, mb in results):,.0f} MB")for mode in ("send", "windows"):
print(subprocess.run([sys.executable, "transfer.py", mode], capture_output=True, text=True).stdout.strip())send 1,354,444 rows 2.6 s largest worker 562 MB
windows 338,611 rows 0.5 s largest worker 259 MB
Sending the frame means four copies of 338,611 geometries pickled to WKB, pushed through a pipe and rebuilt — and the work being done with them is len(). Reading windows means each worker loads only its own quarter.
Sometimes the answer is not parallelism
import time
import geopandas as gpd
import numpy as np
gdf = gpd.read_file("data/parcels.gpkg").to_crs(27700)
t0 = time.perf_counter()
areas_loop = gdf.geometry.apply(lambda g: g.area) # Python per feature
loop_s = time.perf_counter() - t0
t0 = time.perf_counter()
areas_vec = gdf.geometry.area # vectorised in GEOS
vec_s = time.perf_counter() - t0
print(f"apply : {loop_s:6.2f} s")
print(f"vectorised : {vec_s:6.2f} s ({loop_s/vec_s:.0f}× faster, no parallelism at all)")apply : 0.45 s
vectorised : 0.02 s (24× faster, no parallelism at all)
Vectorising usually beats parallelising a slow implementation, and it is simpler. Reach for parallelism after the obvious single-threaded wins, not before.
And sometimes the answer is a database
import geopandas as gpd
from sqlalchemy import create_engine
engine = create_engine("postgresql+psycopg://gis@db.internal/gis")
# PostGIS runs this with its own parallel workers, against indexed data
counts = gpd.pd.read_sql("""
SELECT w.ward_name, count(b.id) AS buildings
FROM wards w LEFT JOIN buildings b ON ST_Within(b.geom, w.geom)
GROUP BY w.ward_name
""", engine)A spatial join across millions of rows is a solved problem in PostGIS, with indexes and a parallel query planner. Moving the data to Python to parallelise it by hand is usually the slower path.
Scaling tests on Birmingham's parcels
The data used on this page
HM Land Registry's INSPIRE parcels for Birmingham — 338,611 polygons in British National Grid — tagged with their ONS ward, saved whole as GeoPackage and Parquet and split into one file per ward; chunks is the geometry column in eight slices. For the network test, the 96 band files of sixteen Earth Search Sentinel-2 scenes over Snowdonia, read over HTTPS.
import io
import os
import time
import zipfile
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import geopandas as gpd
import numpy as np
import pyogrio
import requests
pyogrio.set_gdal_config_options({"GML_DOWNLOAD_SCHEMA": "NO"})
Path("data/wards").mkdir(parents=True, exist_ok=True)
if not Path("data/parcels.parquet").exists():
reply = requests.Session().get("https://use-land-property-data.service.gov.uk/datasets/inspire/download/"
"Birmingham.zip", timeout=600)
reply.raise_for_status()
zipfile.ZipFile(io.BytesIO(reply.content)).extract("Land_Registry_Cadastral_Parcels.gml", "downloads")
parcels = gpd.read_file("downloads/Land_Registry_Cadastral_Parcels.gml")[["INSPIREID", "geometry"]]
wards = gpd.read_file(io.BytesIO(requests.get(
"https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/"
"Wards_December_2024_Boundaries_UK_BGC/FeatureServer/0/query",
params={"where": "LAD24NM='Birmingham'", "outFields": "WD24CD,WD24NM", "outSR": 27700, "f": "geojson"},
timeout=120).content))
centres = parcels.assign(geometry=parcels.representative_point())
parcels["ward"] = gpd.sjoin(centres, wards, predicate="within", how="left")["WD24NM"].values
parcels.to_file("data/parcels.gpkg")
parcels.to_parquet("data/parcels.parquet")
for ward, part in parcels.groupby("ward"):
part.to_file(f"data/wards/{ward.replace(' ', '_').replace('&', 'and')}.gpkg")
parcels = gpd.read_parquet("data/parcels.parquet")
edges = np.linspace(0, len(parcels), 9).astype(int)
chunks = [parcels.geometry.iloc[a:b] for a, b in zip(edges[:-1], edges[1:])]
items = requests.get("https://earth-search.aws.element84.com/v1/search", timeout=60, params={
"collections": "sentinel-2-l2a", "limit": 16, "bbox": "-4.1,53.0,-4.0,53.1",
"datetime": "2025-06-01T00:00:00Z/2025-09-30T23:59:59Z"}).json()["features"]
remote_rasters = [f"/vsicurl/{item['assets'][band]['href']}" for item in items
for band in ("blue", "green", "red", "nir", "swir16", "swir22")]
print(f"{len(parcels):,} parcels, {parcels['ward'].nunique()} ward files; {len(remote_rasters)} remote rasters; "
f"{os.cpu_count()} CPUs, load {os.getloadavg()[0]:.1f}")338,611 parcels, 69 ward files; 96 remote rasters; 24 CPUs, load 0.3
Example 1: pick the executor from the workload
A tempting rule is to time one item and compare CPU time with wall time. Both kinds of CPU-bound work pass that test, though — and only one of them scales with threads:
def cpu_ratio(fn, item):
t0, c0 = time.perf_counter(), time.process_time()
fn(item)
wall, cpu = time.perf_counter() - t0, time.process_time() - c0
return cpu / wall
def thread_speedup(fn, items, workers=4):
t0 = time.perf_counter()
for item in items[:workers]:
fn(item)
serial = time.perf_counter() - t0
t0 = time.perf_counter()
with ThreadPoolExecutor(workers) as pool:
list(pool.map(fn, items[:workers]))
return serial / (time.perf_counter() - t0)
for label, fn in [("buffer (GEOS)", buffer_chunk), ("apply (Python)", area_by_loop)]:
print(f"{label:15} cpu/wall {cpu_ratio(fn, chunks[0]):.2f} 4-thread speed-up {thread_speedup(fn, chunks):.1f}×")buffer (GEOS) cpu/wall 1.00 4-thread speed-up 2.8×
apply (Python) cpu/wall 1.00 4-thread speed-up 0.4×
Both are CPU-bound by the ratio; only the GEOS work speeds up on threads. So decide from the thing you care about: try two or four threads on a few items, and use threads if they help and processes if they do not. It costs a few extra calls and it measures the answer directly.
Example 2: worker initialiser that tames native threads
%%writefile init_workers.py
from concurrent.futures import ProcessPoolExecutor
import multiprocessing as mp
import os
def init_worker():
for var in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS", "GDAL_NUM_THREADS"):
os.environ[var] = "1"
def report(_):
return os.getpid(), os.environ.get("OMP_NUM_THREADS")
if __name__ == "__main__":
with ProcessPoolExecutor(max_workers=4, initializer=init_worker) as pool:
seen = set(pool.map(report, range(40)))
print(f"{len({pid for pid, _ in seen})} workers, OMP_NUM_THREADS in each: {sorted({v for _, v in seen})}")
with mp.Pool(4, initializer=init_worker, maxtasksperchild=10) as pool: # recycling: use Pool
pids = pool.map(report, range(40), chunksize=1)
print(f"Pool with maxtasksperchild=10: 40 tasks in {len({pid for pid, _ in pids})} worker processes")print(subprocess.run([sys.executable, "init_workers.py"], capture_output=True, text=True, timeout=120).stdout)2 workers, OMP_NUM_THREADS in each: ['1']
Pool with maxtasksperchild=10: 40 tasks in 4 worker processes
Recycling workers caps any per-worker memory growth. Do it with multiprocessing.Pool(maxtasksperchild=...): the ProcessPoolExecutor(max_tasks_per_child=...) equivalent deadlocked on Python 3.14.4 as soon as a worker needed replacing, as the parallel batch fix page measures.
Example 3: a scaling test before committing
def scaling_test(fn, items, max_workers=8):
baseline = None
for n in [1, 2, 4, max_workers]:
t0 = time.perf_counter()
with ThreadPoolExecutor(max_workers=n) as pool:
list(pool.map(fn, items))
elapsed = time.perf_counter() - t0
baseline = baseline or elapsed
print(f"{n:>2} workers: {elapsed:6.2f} s speed-up {baseline/elapsed:4.1f}× "
f"efficiency {baseline/elapsed/n:4.0%}")
scaling_test(buffer_chunk, chunks) 1 workers: 6.84 s speed-up 1.0× efficiency 100%
2 workers: 3.36 s speed-up 2.0× efficiency 102%
4 workers: 2.08 s speed-up 3.3× efficiency 82%
8 workers: 1.25 s speed-up 5.5× efficiency 68%
Efficiency below about 50% means the parallelism is not paying for itself — usually contention, oversubscription or transfer cost.
Example 4: Dask, when the dataset does not fit
import dask_geopandas as dgpd
ddf = dgpd.read_parquet("data/parcels.parquet", npartitions=16)
ddf = ddf.to_crs(27700)
ddf["area_m2"] = ddf.geometry.area
result = ddf.groupby("ward")["area_m2"].sum().compute()
print((result / 1e4).round(0).sort_values(ascending=False).head(5).to_string())ward
Sutton Walmley & Minworth 1595.0
Sutton Roughley 830.0
Perry Barr 694.0
Edgbaston 668.0
Bartley Green 520.0
dask-geopandas partitions the data, builds a task graph and runs it across threads or processes. It is the right tool when the dataset exceeds memory and the operations are the standard ones; hand-rolled parallelism remains simpler for whole-file batches. Here the whole thing fits in memory, so this only shows the shape of the code.
What the GIL protects, and what parallelism costs
Two ideas explain almost every parallel-GIS outcome: what the GIL protects, and what parallelism costs.
The GIL serialises the interpreter, not the machine. Any C extension that is not touching Python objects can release it, and the geospatial stack does so extensively: Shapely 2 around GEOS calls, GDAL during I/O and format decoding, NumPy for array arithmetic. Consequently a workload dominated by buffer, intersection, to_crs or file reading really does scale with threads, which is the opposite of the folk wisdom about Python. A workload dominated by per-feature Python — apply, dict building, string handling — does not, because that code holds the lock the entire time.
Processes sidestep the lock by having one interpreter each, and they pay for it three times: start-up (tens to hundreds of milliseconds each, more with spawn), memory (each worker holds its own copy of everything it touches), and transfer (arguments and results are pickled). For file-sized units of work those costs vanish into the noise. For fine-grained work they dominate, which is exactly why the naïve ProcessPoolExecutor version of a chunked geometry operation can be slower than the serial loop.
Oversubscription is the third factor and the least obvious. GDAL, BLAS and PROJ each maintain their own thread pools sized to the machine, so four processes can quietly become thirty-two threads. Cores then spend their time switching contexts and invalidating caches. Setting OMP_NUM_THREADS=1 and its siblings inside workers is not a micro-optimisation; it is often the difference between a speed-up and a slow-down.
Which leads to the practical ordering. First make the single-threaded version fast: vectorise instead of looping, narrow the read with columns= and where=, use the spatial index. Then, if it is still too slow, measure the CPU-to-wall ratio to see what you are waiting for. Use threads for I/O and GEOS-heavy work, processes for whole-file batches, a database for large joins and aggregations, and Dask when the data does not fit at all. And measure the result — parallel code that is not faster is just more complicated.
Fork, spawn and free-threaded Python
forkis unsafe with GDAL and GEOS: Native libraries can hold locks owned by threads that do not survive the fork. Usespawn.- The
__main__guard is mandatory: Withspawn, children re-import your module; without the guard they re-run it. - Free-threaded Python (3.13t+): Experimental builds remove the GIL. Check that every geospatial wheel you depend on is built for it before planning around it.
asynciodoes not help CPU work: It is a concurrency model for I/O; geometry operations still block the loop.- Threads and PyQGIS do not mix: Qt objects belong to their creating thread. Use one process per task instead.
- Shared memory is possible:
multiprocessing.shared_memoryavoids copying large NumPy arrays, but geometry arrays need re-wrapping — usually not worth it. - Reading is parallel-safe, writing usually is not: Several processes appending to one GeoPackage lost rows without an error in the parallel batch fix page's test.
Internal links
- How to Speed Up Batch GIS Jobs with Parallel Processing in Python
- Parallel GIS Batch Job Hangs or Crashes with multiprocessing: How to Fix It
- How to Speed Up GeoPandas: Tips for Large Datasets
- Spatial Indexes Explained: R-trees and Why Spatial Joins Are Fast
- How to Process a Very Large GeoPackage in Chunks with Python
- Batch GIS Job Slows Down and Runs Out of Memory: How to Fix It
FAQ
Does the GIL make threads useless in Python GIS?
No. GEOS, GDAL and NumPy release the GIL while they work, so geometry operations and file I/O do scale with threads. Only per-feature Python code is serialised.
When should I use processes instead of threads?
When the work is dominated by Python-level code, or when each unit is a whole file. Processes cost start-up, memory and pickling, so the unit of work needs to be large enough to absorb that.
Why did parallel processing make my job slower?
Most often oversubscription — each worker starting its own native thread pool — or the cost of pickling GeoDataFrames between processes. Set OMP_NUM_THREADS=1 in workers and pass paths, not data.
How many workers should I use?
For CPU work, about the number of physical cores, and check that memory allows it. For I/O work, more than the core count is fine — 16 to 32 threads is common for reading many small files.
Is asyncio useful for GIS work?
Only for network I/O — fetching many tiles or API responses. It does nothing for geometry computation, which blocks the event loop.
What about Dask?
dask-geopandas is the right tool when the dataset does not fit in memory and you want the standard operations parallelised for you. For whole-file batches, a process pool is simpler.
How do I know whether my code is CPU-bound?
Compare time.process_time() with time.perf_counter() over the same call. A ratio near 1 means CPU-bound; much less means it is waiting on I/O.