🔧 Fix Batch processing Performance & scale Batch Processing · step 25 of 26 Updated

Batch GIS Job Slows Down and Runs Out of Memory: How to Fix It

Problem statement

The first fifty files fly past. By file two hundred each one takes four times as long, and somewhere after that the process dies:

[198/640] tiles_198.gpkg   12.4s
[199/640] tiles_199.gpkg   13.1s
Killed

Killed with no traceback is the Linux OOM killer. On Windows you get a MemoryError, and in a notebook the kernel simply restarts. Either way, the important detail is that no single file is too big — the accumulation is. Memory that should have been released after each iteration is still referenced somewhere.

Common causes:

  • results are appended to a list and concatenated only at the end
  • a cache, dictionary, or functools.lru_cache keyed on file path grows without bound
  • each iteration builds a spatial index or a plot that is never closed
  • exception objects hold a reference to the frame that raised them, keeping large locals alive
  • worker processes in a pool are never recycled, so each accumulates its own leak
  • the loop reads whole files when it only needs a bounding box or a few columns

The distinguishing symptom is the slowdown before the crash: as the resident set grows, the allocator and the OS page cache work harder, and garbage collection has more to scan. Constant memory means constant speed; growing memory shows up as a curve.

Quick answer

To stop a batch from growing without bound:

  1. write each result to disk inside the loop instead of collecting frames in a list
  2. delete large locals at the end of each iteration and let refcounting free them
  3. close every matplotlib figure with plt.close(fig) — never rely on plt.close() alone
  4. read only the columns and rows you need (columns=, bbox=, where=)
  5. measure per-iteration memory so you can prove the leak is gone
from pathlib import Path
import gc
import geopandas as gpd

out = Path("data/out"); out.mkdir(parents=True, exist_ok=True)

for i, path in enumerate(sorted(Path("data/raw").glob("*.gpkg")), start=1):
    gdf = gpd.read_file(path, columns=["id", "class", "geometry"])
    result = gdf.dissolve(by="class", as_index=False)
    result.to_file(out / f"{path.stem}_dissolved.gpkg", driver="GPKG")
    del gdf, result                   # drop the references now, not at loop end
    if i % 50 == 0:
        gc.collect()                  # collect cycles occasionally, not every iteration

Writing inside the loop is the change that matters. A list of 640 GeoDataFrames is 640 datasets in RAM at once; a file on disk is zero.

Where the memory goes

Bar chart of memory growth across a batch run with accumulate versus stream.
Accumulating results grows linearly with the file count; streaming stays flat.

Step-by-step solution

Triage table mapping memory growth causes to fixes.
Five ways a batch leaks, and the one-line fix for each.

Measure before you guess

Add a memory reading per iteration. Two lines will tell you whether you have a leak or a single oversized file.

import os, psutil
proc = psutil.Process(os.getpid())

for i, path in enumerate(files, start=1):
    process_one(path)
    if i % 10 == 0:
        rss = proc.memory_info().rss / 1_048_576
        print(f"[{i}/{len(files)}] rss={rss:.0f} MB", flush=True)

A flat line with occasional spikes means one big file. A staircase that never comes down means accumulation, and the slope tells you how many iterations you have left before the kill.

Without psutil, tracemalloc gives you the Python-side allocation totals and, crucially, where they were allocated:

import tracemalloc
tracemalloc.start()
snapshot_a = tracemalloc.take_snapshot()
# ... run 50 iterations ...
snapshot_b = tracemalloc.take_snapshot()
for stat in snapshot_b.compare_to(snapshot_a, "lineno")[:10]:
    print(stat)

Stream results to disk instead of collecting them

The single most common cause is a list that holds every result until the end.

# leaks: every frame stays resident until the concat
frames = []
for path in files:
    frames.append(gpd.read_file(path))
merged = pd.concat(frames)

# streams: one frame resident at a time
for path in files:
    gdf = gpd.read_file(path)
    gdf.to_file("data/out/merged.gpkg", layer="all", driver="GPKG", mode="a")

If you genuinely need one merged dataset, append into a GeoPackage layer as above, or write per-file Parquet and let a query engine read the folder as one table. Both keep peak memory at the size of the largest single input.

Read less in the first place

Most batch steps use a fraction of what they load. Every reader supports narrowing the read.

import geopandas as gpd

# only the columns you use — attribute tables are often larger than the geometry
gdf = gpd.read_file(path, columns=["id", "class", "geometry"])

# only the area you care about
gdf = gpd.read_file(path, bbox=(-3.1, 55.9, -3.0, 56.0))

# only matching rows, pushed down to the driver (pyogrio engine)
gdf = gpd.read_file(path, where="class = 'residential'")

# a fixed slice, for testing the loop cheaply
gdf = gpd.read_file(path, rows=1000)

columns= and where= are evaluated by the driver, so the rows never enter Python at all — this is far more effective than filtering after the read.

Drop references at the end of each iteration

CPython frees an object as soon as its reference count hits zero. A variable assigned in the loop body stays bound until it is reassigned on the next iteration, which means the previous frame is alive while the next one is being built — a 2× peak.

for path in files:
    gdf = gpd.read_file(path)
    result = expensive_step(gdf)
    result.to_file(out / f"{path.stem}.gpkg", driver="GPKG")
    del gdf, result          # peak is now one frame, not two

Better still, put the body in a function. Locals disappear when the function returns, and the intent is clearer than a del.

def process_one(path: Path) -> None:
    gdf = gpd.read_file(path)
    expensive_step(gdf).to_file(out / f"{path.stem}.gpkg", driver="GPKG")

for path in files:
    process_one(path)

Close figures, datasets and connections

Anything with an open/close pair holds native memory that Python's garbage collector cannot see.

import matplotlib
matplotlib.use("Agg")            # no GUI backend in a batch job
import matplotlib.pyplot as plt

for path in files:
    fig, ax = plt.subplots(figsize=(8, 8))
    gpd.read_file(path).plot(ax=ax)
    fig.savefig(out / f"{path.stem}.png", dpi=150)
    plt.close(fig)               # without this, every figure stays in pyplot's registry

The same applies to rasters and databases — use context managers so they close even when a step raises:

import rasterio

with rasterio.open(path) as src:
    band = src.read(1)           # closed on exit, even on exception

Hold onto error messages, not exception objects

A caught exception keeps a traceback, and a traceback keeps every frame's locals alive — including the 2 GB GeoDataFrame that caused the problem.

# leaks the failing frame's locals into the results list
except Exception as exc:
    failures.append(exc)

# keeps only text
except Exception as exc:
    failures.append(f"{path.name}: {type(exc).__name__}: {exc}")

Python clears the exc name at the end of the except block for this reason, but storing the object elsewhere defeats that.

Recycle pool workers

In a ProcessPoolExecutor, a leak inside the worker function accumulates for the life of the worker. Recycling processes after a fixed number of tasks caps the damage.

from concurrent.futures import ProcessPoolExecutor

with ProcessPoolExecutor(max_workers=4, max_tasks_per_child=25) as pool:
    list(pool.map(process_one, files))

max_tasks_per_child requires Python 3.11+; on older versions, multiprocessing.Pool(maxtasksperchild=25) does the same job.

Code examples

Example 1: a chunked, constant-memory batch 

from pathlib import Path
import geopandas as gpd
import psutil, os

SRC, OUT = Path("data/raw"), Path("data/out")
OUT.mkdir(parents=True, exist_ok=True)
proc = psutil.Process(os.getpid())

def process_one(path: Path) -> int:
    gdf = gpd.read_file(path, columns=["id", "class", "geometry"])
    out = gdf[gdf.geometry.notna() & gdf.geometry.is_valid]
    out.to_file(OUT / f"{path.stem}.gpkg", driver="GPKG")
    return len(out)

total = 0
files = sorted(SRC.glob("*.gpkg"))
for i, path in enumerate(files, start=1):
    total += process_one(path)
    if i % 25 == 0:
        rss = proc.memory_info().rss / 1_048_576
        print(f"[{i}/{len(files)}] {total} features, rss={rss:.0f} MB", flush=True)

Example 2: process a huge layer in row chunks

When a single file is the problem, read it in slices with pyogrio rather than all at once.

import pyogrio
import geopandas as gpd

path = "data/raw/national_parcels.gpkg"
info = pyogrio.read_info(path)
total = info["features"]
CHUNK = 100_000

for start in range(0, total, CHUNK):
    gdf = gpd.read_file(path, engine="pyogrio", skip_features=start, max_features=CHUNK)
    gdf["area_m2"] = gdf.to_crs(3857).area
    mode = "w" if start == 0 else "a"
    gdf.to_file("data/out/parcels_area.gpkg", layer="parcels", driver="GPKG", mode=mode)
    print(f"{start + len(gdf)}/{total}")
    del gdf

Example 3: raster batches with windowed reads 

import rasterio
from rasterio.windows import Window

with rasterio.open("data/raw/dem.tif") as src:
    profile = src.profile
    with rasterio.open("data/out/dem_scaled.tif", "w", **profile) as dst:
        for ji, window in src.block_windows(1):
            block = src.read(1, window=window)
            dst.write((block * 0.3048).astype(profile["dtype"]), 1, window=window)

Block-windowed I/O keeps a single tile in memory regardless of the raster's size, which is the raster equivalent of streaming rows.

Example 4: find the leak with tracemalloc 

import tracemalloc
from pathlib import Path

tracemalloc.start(10)
files = sorted(Path("data/raw").glob("*.gpkg"))

for i, path in enumerate(files, start=1):
    process_one(path)
    if i == 20:
        base = tracemalloc.take_snapshot()
    if i == 60:
        diff = tracemalloc.take_snapshot().compare_to(base, "traceback")
        for stat in diff[:5]:
            print(f"{stat.size_diff/1_048_576:.1f} MB in {stat.count_diff} blocks")
            for line in stat.traceback.format():
                print("   ", line)
        break

Comparing two snapshots taken mid-run points at the exact line that is still holding memory.

Explanation

Peak memory in a loop is determined by what remains reachable at the moment of highest allocation. In an accumulating loop that is every result produced so far, plus the one being built. In a streaming loop it is one input plus one output. The file count therefore appears in the first formula and not the second — which is why the crash always arrives at "some large number of files" rather than on a specific file.

Vertical steps of a constant-memory loop: read narrowed, transform, write, release.
Read narrow, transform, write, release — the loop's memory profile is then flat by construction.

CPython frees objects immediately when their reference count reaches zero; the cycle collector only handles objects that reference each other. That has two practical consequences. First, gc.collect() rarely fixes a GeoDataFrame leak — if memory does not drop after del, something still holds a reference, and collecting cycles will not change that. Second, the things that do need explicit attention are native resources: GDAL datasets, matplotlib figures, database connections. Python's view of them is a small handle, so the interpreter feels no pressure to release the megabytes behind it.

The slowdown that precedes the crash is a useful diagnostic. Growing memory means more page faults, worse cache locality, and longer garbage-collection scans, so per-file time rises smoothly. If time per file is flat right up to the kill, the problem is more likely one enormous input than an accumulation.

Finally, reading less beats freeing more. A where= clause or a columns= list pushes the filter into GDAL, so the excluded data is never materialised as Python objects at all. Loading a 40-column parcels layer to compute one area column is the most common avoidable cost in a GIS batch.

Edge cases or notes

  • Killed with no traceback is the OOM killer: Check dmesg -T | grep -i oom on Linux to confirm. Python never saw the error, so no except could have caught it.
  • Freed memory is not always returned to the OS: The allocator may keep arenas for reuse, so RSS can plateau rather than drop. What matters is that it stops growing.
  • Notebooks keep every output alive: Out[…] and _ hold references to previous cell results. Restart the kernel before measuring, and avoid displaying large frames in a loop.
  • copy() is cheaper than you think, views are not: Slicing a GeoDataFrame can keep the parent alive through a view. Call .copy() on the slice you keep and drop the parent.
  • Parallelism multiplies peak memory: Four workers means four copies of the per-file peak. Size max_workers against available RAM, not core count.
  • Dissolve, overlay and sjoin can spike well above input size: Intermediate results in these operations are often several times the inputs. Chunk by region if a single call is the peak.

FAQ

Why does my batch die after 200 files when each file is only 50 MB?

Because something is keeping the earlier results alive — usually a list of frames, a cache, or stored exception objects. Fifty MB times 200 is 10 GB, which is a very ordinary way to exhaust RAM without any single file being large.

Does calling gc.collect() in the loop fix a leak?

Almost never. It collects reference cycles, but a GeoDataFrame held by a list is not a cycle — it is simply still referenced. Find and remove the reference; call gc.collect() occasionally at most.

Why did memory not drop after del gdf?

Either another name still refers to the object (a slice, a list entry, a closure), or the allocator kept the memory for reuse rather than returning it to the OS. Check with sys.getrefcount() or a tracemalloc snapshot diff.

How do I process one file that is too big for RAM?

Read it in slices with skip_features/max_features via pyogrio, or filter at read time with bbox=/where=. For rasters, use windowed reads over src.block_windows().

Is Killed the same as MemoryError?

No. MemoryError is raised by Python when an allocation fails and can be caught. Killed means the OS terminated the process, usually the Linux OOM killer, and no Python code runs at that point.

Should I use multiprocessing to reduce memory?

It does not reduce memory — each worker has its own full copy of the per-file peak, so total usage multiplies. It helps only if the leak was per-worker and you recycle workers with max_tasks_per_child.

Why do my matplotlib maps leak in a batch?

pyplot keeps a registry of every figure until it is closed. Call plt.close(fig) after each save and set the non-interactive Agg backend at the top of the script.