Batch GIS Job Slows Down and Runs Out of Memory: How to Fix It
On this page
Problem statement
The first files fly past, and the process grows with every one of them. This is a real run — a loop that draws each of 60 countries' provinces to a PNG and never closes the figure (the full test is under step 5):
close=False [20/60] rss + 99 MB, 20 figures open
close=False [40/60] rss + 188 MB, 40 figures open
close=False [60/60] rss + 300 MB, 60 figures openNothing has failed yet. At five megabytes a figure, a few thousand files is more memory than most machines have, and then the process dies — on Linux with a bare Killed.
Killed with no traceback is the Linux OOM killer. On Windows you get a MemoryError, and in a notebook the kernel simply restarts. Either way, the important detail is that no single file is too big — the accumulation is. Memory that should have been released after each iteration is still referenced somewhere.
Where the accumulation usually comes from:
- results are appended to a list and concatenated only at the end
- a cache, dictionary, or
functools.lru_cachekeyed on file path grows without bound - each iteration builds a spatial index or a plot that is never closed
- exception objects hold a reference to the frame that raised them, keeping large locals alive
- worker processes in a pool are never recycled, so each accumulates its own leak
- the loop reads whole files when it only needs a bounding box or a few columns
The distinguishing symptom is the slowdown before the crash: as the resident set grows, the allocator and the OS page cache work harder, and garbage collection has more to scan. Constant memory means constant speed; growing memory shows up as a curve.
Quick answer
To stop a batch from growing without bound:
- write each result to disk inside the loop instead of collecting frames in a list
- delete large locals at the end of each iteration and let refcounting free them
- close every matplotlib figure with
plt.close(fig)— never rely onplt.close()alone - read only the columns and rows you need (
columns=,bbox=,where=) - measure per-iteration memory so you can prove the leak is gone
from pathlib import Path
import gc
import geopandas as gpd
out = Path("data/out"); out.mkdir(parents=True, exist_ok=True)
for i, path in enumerate(sorted(Path("data/raw").glob("*.gpkg")), start=1):
gdf = gpd.read_file(path, columns=["id", "class", "geometry"])
result = gdf.dissolve(by="class", as_index=False)
result.to_file(out / f"{path.stem}_dissolved.gpkg", driver="GPKG")
del gdf, result # drop the references now, not at loop end
if i % 50 == 0:
gc.collect() # collect cycles occasionally, not every iteration
print(f"{i} files, rss {rss():.0f} MB")251 files, rss 280 MB
Writing inside the loop is the change that matters. A list of 640 GeoDataFrames is 640 datasets in RAM at once; a file on disk is zero.
Where the memory goes
Measure, stream, and let go of each iteration
Measure before you guess
Add a memory reading per iteration. Two lines will tell you whether you have a leak or a single oversized file.
import os, psutil
proc = psutil.Process(os.getpid())
for i, path in enumerate(files, start=1):
process_one(path)
if i % 50 == 0:
rss_mb = proc.memory_info().rss / 1_048_576
print(f"[{i}/{len(files)}] rss={rss_mb:.0f} MB", flush=True)[50/251] rss=280 MB
[100/251] rss=280 MB
[150/251] rss=280 MB
[200/251] rss=289 MB
[250/251] rss=292 MB
A flat line with occasional spikes means one big file. A staircase that never comes down means accumulation, and the slope tells you how many iterations you have left before the kill.
Without psutil, tracemalloc gives you the Python-side allocation totals and, crucially, where they were allocated:
import tracemalloc
tracemalloc.start()
snapshot_a = tracemalloc.take_snapshot()
kept = [gpd.read_file(path) for path in files[:50]] # 50 iterations of a loop that keeps its frames
snapshot_b = tracemalloc.take_snapshot()
for stat in snapshot_b.compare_to(snapshot_a, "lineno")[:5]:
frame = stat.traceback[0]
print(f"{stat.size_diff / 1_048_576:6.1f} MB {frame.filename.split('site-packages/')[-1]}:{frame.lineno}")
tracemalloc.stop()
del kept 9.4 MB pyogrio/raw.py:200
0.8 MB pandas/core/internals/blocks.py:650
0.6 MB pandas/core/construction.py:616
0.4 MB pandas/core/internals/blocks.py:645
0.3 MB pandas/core/internals/managers.py:2431
Stream results to disk instead of collecting them
The single most common cause is a list that holds every result until the end.
import pandas as pd
# leaks: every frame stays resident until the concat
start = rss()
frames = []
for path in files:
frames.append(gpd.read_file(path))
merged = pd.concat(frames)
print(f"accumulate + concat: rss +{rss() - start:.0f} MB")
del frames, merged
print(f"after del: rss +{rss() - start:.0f} MB")
# streams: one frame resident at a time
start = rss()
for path in files:
gdf = gpd.read_file(path)
gdf.to_file("data/out/merged.gpkg", layer="all", driver="GPKG", mode="a")
print(f"stream: rss +{rss() - start:.0f} MB")accumulate + concat: rss +26 MB
after del: rss +26 MB
stream: rss +1 MB
If you genuinely need one merged dataset, append into a GeoPackage layer as above, or write per-file Parquet and let a query engine read the folder as one table. Both keep peak memory at the size of the largest single input.
Read less in the first place
Most batch steps use a fraction of what they load. Every reader supports narrowing the read.
import geopandas as gpd
path = raw / "GBR.gpkg"
print("everything: ", gpd.read_file(path).shape)
# only the columns you use — attribute tables are often larger than the geometry
gdf = gpd.read_file(path, columns=["id", "class", "geometry"])
print("3 columns: ", gdf.shape)
# only the area you care about
gdf = gpd.read_file(path, bbox=(-3.1, 55.9, -3.0, 56.0))
print("bbox: ", gdf.shape, gdf["name"].tolist())
# only matching rows, pushed down to the driver (pyogrio engine)
gdf = gpd.read_file(path, where="class = 'Unitary Authority'")
print("where: ", gdf.shape)
# a fixed slice, for testing the loop cheaply
gdf = gpd.read_file(path, rows=10)
print("rows=10: ", gdf.shape)everything: (232, 122)
3 columns: (232, 3)
bbox: (3, 122) ['Midlothian', 'Edinburgh', 'East Lothian']
where: (50, 122)
rows=10: (10, 122)
columns= and where= are evaluated by the driver, so the rows never enter Python at all — this is far more effective than filtering after the read.
Drop references at the end of each iteration
CPython frees an object as soon as its reference count hits zero. A variable assigned in the loop body stays bound until it is reassigned on the next iteration, which means the previous frame is alive while the next one is being built — a 2× peak.
for path in files:
gdf = gpd.read_file(path)
result = expensive_step(gdf)
result.to_file(out / f"{path.stem}.gpkg", driver="GPKG")
del gdf, result # peak is now one frame, not twoBetter still, put the body in a function. Locals disappear when the function returns, and the intent is clearer than a del.
def process_one(path: Path) -> None:
gdf = gpd.read_file(path)
expensive_step(gdf).to_file(out / f"{path.stem}.gpkg", driver="GPKG")
for path in files:
process_one(path)Close figures, datasets and connections
Anything with an open/close pair holds native memory that Python's garbage collector cannot see.
import matplotlib
matplotlib.use("Agg") # no GUI backend in a batch job
import matplotlib.pyplot as plt
Path("data/png").mkdir(exist_ok=True)
for close in (True, False):
start = rss()
for i, path in enumerate(files[:60], start=1):
fig, ax = plt.subplots(figsize=(8, 8))
gpd.read_file(path).plot(ax=ax)
fig.savefig(f"data/png/{path.stem}.png", dpi=150)
if close:
plt.close(fig) # without this, every figure stays in pyplot's registry
if i % 20 == 0:
print(f"close={close!s:5} [{i}/60] rss +{rss() - start:4.0f} MB, {len(plt.get_fignums())} figures open")
plt.close("all")close=True [20/60] rss + 78 MB, 0 figures open
close=True [40/60] rss + 189 MB, 0 figures open
close=True [60/60] rss + 60 MB, 0 figures open
close=False [20/60] rss + 99 MB, 20 figures open
close=False [40/60] rss + 188 MB, 40 figures open
close=False [60/60] rss + 300 MB, 60 figures open
Without close, all 60 figures stay open and memory only climbs — 300 MB by the end, five megabytes a figure. With it, memory still swings, up to 189 MB in mid-run, because the allocator keeps freed memory for reuse rather than handing it back; but nothing is held, and it ends where the next file can reuse it.
The same applies to rasters and databases — use context managers so they close even when a step raises:
import rasterio
with rasterio.open(dem) as src:
band = src.read(1) # closed on exit, even on exception
print(band.shape, "read; dataset closed:", src.closed)(1200, 800) read; dataset closed: True
Hold onto error messages, not exception objects
A caught exception keeps a traceback, and a traceback keeps every frame's locals alive — including the 2 GB GeoDataFrame that caused the problem.
# leaks the failing frame's locals into the results list
except Exception as exc:
failures.append(exc)
# keeps only text
except Exception as exc:
failures.append(f"{path.name}: {type(exc).__name__}: {exc}")Python clears the exc name at the end of the except block for this reason, but storing the object elsewhere defeats that.
Recycle pool workers
In a ProcessPoolExecutor, a leak inside the worker function accumulates for the life of the worker. Recycling processes after a fixed number of tasks caps the damage.
%%writefile jobs.py
from pathlib import Path
import geopandas as gpd
def process_one(path):
gdf = gpd.read_file(path, columns=["id", "class", "geometry"])
gdf.dissolve(by="class", as_index=False).to_file(Path("data/out") / f"{Path(path).stem}.gpkg", driver="GPKG")from concurrent.futures import ProcessPoolExecutor
import jobs # workers import their function; a notebook cannot be imported
# a fresh pool per 100 files: every worker's memory is returned when its pool closes
for start in range(0, len(files), 100):
with ProcessPoolExecutor(max_workers=4) as pool:
list(pool.map(jobs.process_one, files[start:start + 100]))
print(f"{len(files)} files through {-(-len(files) // 100)} pools; "
f"{sum((out / f.name).exists() for f in files)} outputs")251 files through 3 pools; 251 outputs
ProcessPoolExecutor(max_tasks_per_child=25) is the standard-library way to recycle workers, and on Python 3.14.4 it did not work here: with four workers and 120 files it completed exactly 100 tasks — four workers × 25 — and then waited indefinitely, because no replacement worker ever started. That happened with both the forkserver and spawn start methods (it refuses fork outright). A pool per chunk gets the same effect — workers that live for a bounded amount of work — with nothing to deadlock.
The older multiprocessing.Pool does recycle correctly on the same Python, and takes a function and an iterable just as pool.map does:
import multiprocessing as mp
import time
start = time.perf_counter()
with mp.Pool(4, maxtasksperchild=25) as pool:
pool.map(jobs.process_one, files[:120])
print(f"Pool(maxtasksperchild=25): 120 files in {time.perf_counter() - start:.1f} s")Pool(maxtasksperchild=25): 120 files in 5.7 s
Constant memory across 251 province files
The data used on this page
Natural Earth's 1:10 million states and provinces (public domain) — 4,596 polygons with 122 attribute columns — split into one GeoPackage per country, as a batch would receive them: 251 files in data/raw/, with ne_id renamed id and the English unit type (Province, District, State …) as class. The chunked-read example uses the 56,600-feature Natural Earth roads layer, and the raster example one Copernicus GLO-90 DEM tile. Memory is measured as the process's resident set size (RSS) with psutil.
import io
import os
import zipfile
from pathlib import Path
import geopandas as gpd
import psutil
import requests
def natural_earth(theme, name, folder=Path("data/src")):
if not (folder / f"{name}.shp").exists():
reply = requests.get(f"https://naciscdn.org/naturalearth/10m/{theme}/{name}.zip", timeout=300)
reply.raise_for_status()
zipfile.ZipFile(io.BytesIO(reply.content)).extractall(folder)
return folder / f"{name}.shp"
provinces = gpd.read_file(natural_earth("cultural", "ne_10m_admin_1_states_provinces"))
raw, out = Path("data/raw"), Path("data/out")
raw.mkdir(parents=True, exist_ok=True)
out.mkdir(parents=True, exist_ok=True)
for country, rows in provinces.groupby("adm0_a3"):
if not (raw / f"{country}.gpkg").exists():
rows.rename(columns={"ne_id": "id", "type_en": "class"}).to_file(raw / f"{country}.gpkg", driver="GPKG")
roads = natural_earth("cultural", "ne_10m_roads")
dem = Path("data/src/dem.tif") # Copernicus GLO-90, the Lake District
if not dem.exists():
name = "Copernicus_DSM_COG_30_N54_00_W004_00_DEM"
dem.write_bytes(requests.get(f"https://copernicus-dem-90m.s3.amazonaws.com/{name}/{name}.tif", timeout=120).content)
files = sorted(raw.glob("*.gpkg"))
proc = psutil.Process(os.getpid())
rss = lambda: proc.memory_info().rss / 1_048_576
def expensive_step(gdf):
return gdf.dissolve(by="class", as_index=False)
def process_one(path):
gdf = gpd.read_file(path, columns=["id", "class", "geometry"])
expensive_step(gdf).to_file(out / f"{Path(path).stem}.gpkg", driver="GPKG")
print(f"{len(provinces):,} provinces, {provinces.shape[1]} columns -> {len(files)} files, "
f"{sum(f.stat().st_size for f in files) / 1e6:.1f} MB")4,596 provinces, 122 columns -> 251 files, 53.7 MB
Example 1: a chunked, constant-memory batch
from pathlib import Path
import geopandas as gpd
import psutil, os
SRC, OUT = Path("data/raw"), Path("data/out")
OUT.mkdir(parents=True, exist_ok=True)
proc = psutil.Process(os.getpid())
def process_one(path: Path) -> int:
gdf = gpd.read_file(path, columns=["id", "class", "geometry"])
out = gdf[gdf.geometry.notna() & gdf.geometry.is_valid]
out.to_file(OUT / f"{path.stem}.gpkg", driver="GPKG")
return len(out)
total = 0
files = sorted(SRC.glob("*.gpkg"))
for i, path in enumerate(files, start=1):
total += process_one(path)
if i % 50 == 0:
rss_mb = proc.memory_info().rss / 1_048_576
print(f"[{i}/{len(files)}] {total} features, rss={rss_mb:.0f} MB", flush=True)[50/251] 748 features, rss=781 MB
[100/251] 1691 features, rss=781 MB
[150/251] 2734 features, rss=785 MB
[200/251] 3536 features, rss=796 MB
[250/251] 4585 features, rss=796 MB
Example 2: process a huge layer in row chunks
When a single file is the problem, read it in slices with pyogrio rather than all at once.
import pyogrio
import geopandas as gpd
path = roads # 56,600 Natural Earth roads
info = pyogrio.read_info(path)
total = info["features"]
CHUNK = 10_000
start_rss = rss()
for start in range(0, total, CHUNK):
gdf = gpd.read_file(path, engine="pyogrio", skip_features=start, max_features=CHUNK)
gdf["length_m"] = gdf.to_crs(8857).length
mode = "w" if start == 0 else "a"
gdf.to_file("data/out/roads_length.gpkg", layer="roads", driver="GPKG", mode=mode)
print(f"{start + len(gdf)}/{total} rss +{rss() - start_rss:.0f} MB")
del gdf10000/56600 rss +28 MB
20000/56600 rss +34 MB
30000/56600 rss +38 MB
40000/56600 rss +48 MB
50000/56600 rss +42 MB
56600/56600 rss +40 MB
Example 3: raster batches with windowed reads
import rasterio
from rasterio.windows import Window
with rasterio.open(dem) as src:
profile = src.profile
print("internal blocks:", src.block_shapes, "for", src.width, "×", src.height)
# one block holds the whole tile, so block_windows would read it all at once: use row strips
strips = [Window(0, row, src.width, min(100, src.height - row)) for row in range(0, src.height, 100)]
with rasterio.open("data/out/dem_ft.tif", "w", **profile) as dst:
for window in strips:
block = src.read(1, window=window)
dst.write((block * 3.28084).astype(profile["dtype"]), 1, window=window) # metres to feet
print(len(strips), "strips of at most", 100 * src.width, "cells")internal blocks: [(2048, 2048)] for 800 × 1200
12 strips of at most 80000 cells
Block-windowed I/O keeps a single tile in memory regardless of the raster's size, which is the raster equivalent of streaming rows.
Example 4: find the leak with tracemalloc
import tracemalloc
from pathlib import Path
tracemalloc.start(10)
files = sorted(Path("data/raw").glob("*.gpkg"))
for i, path in enumerate(files, start=1):
process_one(path)
if i == 20:
base = tracemalloc.take_snapshot()
if i == 60:
diff = tracemalloc.take_snapshot().compare_to(base, "traceback")
for stat in diff[:3]:
frame = stat.traceback[-1] # the most recent frame
where = frame.filename.split("site-packages/")[-1]
print(f"{stat.size_diff/1_048_576:.2f} MB in {stat.count_diff} blocks, "
f"{Path(where).name if where.startswith('/') else where}:{frame.lineno}")
break
tracemalloc.stop()4.62 MB in 1947 blocks, pyogrio/raw.py:200
0.01 MB in 71 blocks, shapely/io.py:200
0.01 MB in 200 blocks, __init__.py:297
Comparing two snapshots taken mid-run points at the exact line that is still holding memory.
What keeps memory reachable in a loop
Peak memory in a loop is determined by what remains reachable at the moment of highest allocation. In an accumulating loop that is every result produced so far, plus the one being built. In a streaming loop it is one input plus one output. The file count therefore appears in the first formula and not the second — which is why the crash always arrives at "some large number of files" rather than on a specific file.
CPython frees objects immediately when their reference count reaches zero; the cycle collector only handles objects that reference each other. That has two practical consequences. First, gc.collect() rarely fixes a GeoDataFrame leak — if memory does not drop after del, something still holds a reference, and collecting cycles will not change that. Second, the things that do need explicit attention are native resources: GDAL datasets, matplotlib figures, database connections. Python's view of them is a small handle, so the interpreter feels no pressure to release the megabytes behind it.
The slowdown that precedes the crash is a useful diagnostic. Growing memory means more page faults, worse cache locality, and longer garbage-collection scans, so per-file time rises smoothly. If time per file is flat right up to the kill, the problem is more likely one enormous input than an accumulation.
Finally, reading less beats freeing more. A where= clause or a columns= list pushes the filter into GDAL, so the excluded data is never materialised as Python objects at all. Loading a 40-column parcels layer to compute one area column is the most common avoidable cost in a GIS batch.
The OOM killer and memory the OS never gets back
Killedwith no traceback is the OOM killer: Checkdmesg -T | grep -i oomon Linux to confirm. Python never saw the error, so noexceptcould have caught it.- Freed memory is not always returned to the OS: The allocator may keep arenas for reuse, so RSS can plateau rather than drop. What matters is that it stops growing.
- Notebooks keep every output alive:
Out[…]and_hold references to previous cell results. Restart the kernel before measuring, and avoid displaying large frames in a loop. copy()is cheaper than you think, views are not: Slicing a GeoDataFrame can keep the parent alive through a view. Call.copy()on the slice you keep and drop the parent.- Parallelism multiplies peak memory: Four workers means four copies of the per-file peak. Size
max_workersagainst available RAM, not core count. - Dissolve, overlay and sjoin can spike well above input size: Intermediate results in these operations are often several times the inputs. Chunk by region if a single call is the peak.
Internal links
- Fixing Memory Errors in GeoPandas When Working with Large Files
- How to Speed Up GeoPandas: Tips for Large Datasets
- How to Batch Process a Folder of GIS Files in Python: The Complete Workflow
- How to Speed Up Batch GIS Jobs with Parallel Processing in Python
- How to Batch Process Rasters with Rasterio in Python
- Parallel GIS Batch Job Hangs or Crashes with multiprocessing: How to Fix It
FAQ
Why does my batch die after 200 files when each file is only 50 MB?
Because something is keeping the earlier results alive — usually a list of frames, a cache, or stored exception objects. Fifty MB times 200 is 10 GB, which is a very ordinary way to exhaust RAM without any single file being large.
Does calling gc.collect() in the loop fix a leak?
Almost never. It collects reference cycles, but a GeoDataFrame held by a list is not a cycle — it is simply still referenced. Find and remove the reference; call gc.collect() occasionally at most.
Why did memory not drop after del gdf?
Either another name still refers to the object (a slice, a list entry, a closure), or the allocator kept the memory for reuse rather than returning it to the OS. Check with sys.getrefcount() or a tracemalloc snapshot diff.
How do I process one file that is too big for RAM?
Read it in slices with skip_features/max_features via pyogrio, or filter at read time with bbox=/where=. For rasters, use windowed reads over src.block_windows().
Is Killed the same as MemoryError?
No. MemoryError is raised by Python when an allocation fails and can be caught. Killed means the OS terminated the process, usually the Linux OOM killer, and no Python code runs at that point.
Should I use multiprocessing to reduce memory?
It does not reduce memory — each worker has its own full copy of the per-file peak, so total usage multiplies. It helps only if the leak was per-worker and you recycle the workers — with multiprocessing.Pool(maxtasksperchild=...), or a fresh pool per chunk.
Why do my matplotlib maps leak in a batch?
pyplot keeps a registry of every figure until it is closed. Call plt.close(fig) after each save and set the non-interactive Agg backend at the top of the script.