Fixing Memory Errors in GeoPandas When Working with Large Files

On this page

Problem statement

Reading 338,611 Birmingham land parcels into GeoPandas peaks at 694 MB from 400.9 MB of GeoJSON and 833 MB from 62.7 MB of Parquet — the smaller file needed more. An overlay with the city's 69 wards takes the same data past 1 GB, and reprojecting the whole layer to about 1.3 GB. On a machine or container with less headroom than that, the job does not reliably stop with MemoryError: under a 2 GB limit the read failed with shapely.errors.GEOSException: std::bad_alloc, and under 1.5 GB the process was killed without a message.

This page measures what each fix actually saves, one strategy per fresh process, on that real dataset. The short version is that the fix is almost always at the read call. Reading one column instead of six cut the peak from 699 MB to 530 MB; reading a 2 km bounding box cut it to what an empty Python with GeoPandas imported already uses.

Quick answer

Make the read narrow, because the peak happens during the read and nothing you drop afterwards lowers it:

import geopandas as gpd

parcels = gpd.read_file(
    "data/parcels.gpkg",
    columns=["INSPIREID"],                    # only the fields the job uses
    bbox=(405000, 285500, 407000, 287500),    # only the study area, via the spatial index
)

What each change saved on the Birmingham parcels (peak resident memory, measured under Code examples):

Change	Peak before	Peak after
`columns=` in the read	699 MB	530 MB
`bbox=` in the read (2 km square)	699 MB	182 MB — the same as importing GeoPandas
filter before `to_crs()`	1,294 MB	875 MB
`sjoin()` instead of `overlay()`	1,043 MB	791 MB
rebind one name instead of making new ones	1,451 MB	1,220 MB

Two things that do not work the way they look: converting GeoJSON to Parquet saves time, not memory (the Parquet read peaked higher, at 833 MB), and del does not hand memory back to the operating system (524 MB loaded, 506 MB after del). And search failed jobs for bad_alloc and Killed as well as MemoryError — under a memory limit all three were the same failure.

Diagnosis at a glance

Cutting memory use at the step where it runs out.

Where it fails, what the file holds, leaner reads and processing

Work down the list — the first two usually solve it on their own.

Every snippet below uses one real, large dataset: HM Land Registry's INSPIRE Index Polygons for Birmingham — 338,611 freehold parcels — with Birmingham's 69 wards from the ONS, set up under Code examples. Each strategy's peak memory is measured there in a separate process, so the numbers quoted here are what that code actually cost.

1) Check where the memory error happens

First, identify the step that fails: reading, processing, or writing.

Reading the file

If the error happens in read_file(), common causes include:

a very large GeoJSON
a shapefile with many fields
complex polygon geometry
an inefficient source format for the job

import geopandas as gpd

gdf = gpd.read_file("data/parcels.geojson")
print(len(gdf))

If this fails immediately, read less data (section 3). It may not fail with MemoryError at all: under a 2 GB limit the full read failed with shapely.errors.GEOSException: std::bad_alloc, and under 1.5 GB it raised MemoryError or was killed without a message. All of them are the same problem.

Processing the data

If the file loads successfully but later fails, the memory spike is usually caused by operations such as to_crs(), sjoin(), overlay(), dissolve(), or buffer().

parcels = gpd.read_file("data/parcels.gpkg")
wards = gpd.read_file("data/wards.gpkg")

result = parcels.overlay(wards, how="intersection")

Writing the output

Large exports can also fail, especially to GeoJSON or shapefile.

result.to_file("output/result.geojson", driver="GeoJSON")

If reading and processing work but export fails, keep intermediate data in a more efficient format and export the final result last.

2) Check file size and structure before loading everything

A large file on disk usually needs more memory after it is parsed into geometry and attribute objects — or less. The parcels are 400.9 MB as GeoJSON, 194.0 MB as GeoPackage and 62.7 MB as Parquet, and every one of them loads into the same GeoDataFrame of roughly 0.6–0.7 GB. Disk size says more about the format than about the memory.

Inspect file size and structure without reading the features

from pathlib import Path
import pyogrio

for name in ["parcels.geojson", "parcels.gpkg", "parcels.parquet"]:
    print(f"{name:16} {Path('data', name).stat().st_size / 1e6:7.1f} MB on disk")
info = pyogrio.read_info("data/parcels.gpkg")
print(info["features"], "features,", info["geometry_type"], "|", list(info["fields"]))

parcels.geojson    400.9 MB on disk
parcels.gpkg       194.0 MB on disk
parcels.parquet     62.7 MB on disk
338611 features, Polygon | ['gml_id', 'INSPIREID', 'LABEL', 'NATIONALCADASTRALREFERENCE', 'VALIDFROM', 'BEGINLIFESPANVERSION']

read_info reads the header, not the features, so it answers "how big is this?" in milliseconds whatever the file size.

Test with a small sample

A small spatial sample helps confirm the workflow before running the full job.

bbox = (405000, 285500, 407000, 287500)      # 2 km square around the city centre, EPSG:27700
sample = gpd.read_file("data/parcels.gpkg", bbox=bbox)
print(len(sample), "parcels in the sample")

2440 parcels in the sample

This reads only what the GeoPackage's spatial index says is in the box.

3) Reduce memory use when reading large spatial files

Read only the columns you need

gdf = gpd.read_file("data/parcels.gpkg", columns=["INSPIREID"])

Pass columns= to the read, not a selection after it. Selecting after the read lowers later processing cost, but the peak has already happened: reading all six columns peaked at 699 MB, reading one at 530 MB.

Use spatial filtering when supported

Bounding box filtering is useful for large workflows, especially when you only need one study area.

gdf = gpd.read_file("data/parcels.gpkg", bbox=(405000, 285500, 407000, 287500))

That read peaked no higher than an empty Python with GeoPandas imported.

Prefer more efficient formats

If you repeatedly use the same large GeoJSON or shapefile, convert it once and reuse the converted version.

gdf = gpd.read_file("data/parcels.geojson")
gdf.to_file("data/parcels.gpkg", driver="GPKG")
gdf.to_parquet("data/parcels.parquet")

The win is time more than memory. The GeoJSON took about 5 s to read, the GeoPackage about 1 s and the Parquet under half a second — but the Parquet read peaked higher, at 833 MB, because Arrow holds its own buffers while the GeoDataFrame is built. Choosing columns= on Parquet barely helped here (791 MB), since the geometry column is most of the data.

4) Reduce memory use during processing

Both are one-line changes, and between them they decide whether the step fits.

Drop unused columns before heavy operations

Before joins or overlays, remove fields you do not need.

parcels = parcels[["INSPIREID", "geometry"]]
wards = wards[["WD24NM", "geometry"]]

Avoid keeping intermediates alive

Every name that still points at an intermediate GeoDataFrame keeps it in memory.

parcels = parcels.to_crs(3857)
parcels = parcels[parcels.area > 100]

Reuse the same variable when earlier intermediate data is no longer needed. The same three steps written as a, b, c peaked at 1,451 MB; rebinding one name peaked at 1,220 MB.

Simplify or clean geometries

If full geometry precision is not required, simplify first.

parcels["geometry"] = parcels.geometry.simplify(tolerance=1, preserve_topology=True)

Repair invalid geometries before expensive operations.

parcels = parcels[parcels.geometry.notna()].copy()
parcels["geometry"] = parcels.geometry.make_valid()

Reproject only when necessary

Do not reproject every layer at the start unless the analysis actually needs it.

parcels = gpd.read_parquet("data/parcels.parquet")
centre = parcels.cx[405000:407000, 285500:287500]
centre_3857 = centre.to_crs(3857)

Reprojecting all 338,611 parcels peaked at about 1.3 GB; filtering to the centre first and reprojecting that kept the peak at the size of the read.

Process data in smaller batches

Split by region, district, tile, or attribute group. If you batch with a district bounding box, expect extra features near the edges unless you clip or filter them afterward.

from pathlib import Path

wards = gpd.read_file("data/wards.gpkg")
Path("temp").mkdir(exist_ok=True)
for code, ward in wards.groupby("WD24CD"):
    subset = gpd.read_file("data/parcels.gpkg", bbox=tuple(ward.total_bounds))
    subset.to_parquet(f"temp/parcels_{code}.parquet")

On Birmingham's wards this reads 1.9 times as many parcels as exist — a bounding box is a rectangle and wards are not — so filter each batch to its ward before combining the results.

5) Use safer approaches for memory-heavy operations

Filter both layers before a spatial join

Reduce both inputs before running the join.

bbox = (405000, 285500, 407000, 287500)
parcels = gpd.read_file("data/parcels.gpkg", bbox=bbox, columns=["INSPIREID"])
wards = gpd.read_file("data/wards.gpkg", bbox=bbox, columns=["WD24NM"])

joined = gpd.sjoin(parcels, wards, predicate="intersects")

Use simpler operations instead of overlay when possible

If you only need a yes/no spatial relationship, a spatial join may be enough. Full overlay() is heavier because it creates new geometries: on the parcels and wards it peaked at 1,043 MB against 791 MB for sjoin, and took four times as long.

Reduce fields before dissolve

Keep only the grouping field and geometry before dissolve().

dissolved = joined[["WD24NM", "geometry"]].dissolve(by="WD24NM")

Peak memory per strategy on Birmingham parcels

The data used on this page

HM Land Registry's INSPIRE Index Polygons for Birmingham (Open Government Licence, with Ordnance Survey rights) — 338,611 freehold parcels — written as GeoPackage, GeoJSON and Parquet, and Birmingham's wards from the ONS December 2024 boundaries.

import io
import os
import zipfile

import geopandas as gpd
import requests

os.makedirs("data", exist_ok=True)
if not os.path.exists("data/parcels.parquet"):
    reply = requests.Session().get("https://use-land-property-data.service.gov.uk/datasets/"
                                   "inspire/download/Birmingham.zip", timeout=600)
    reply.raise_for_status()
    zipfile.ZipFile(io.BytesIO(reply.content)).extract("Land_Registry_Cadastral_Parcels.gml")
    parcels = gpd.read_file("Land_Registry_Cadastral_Parcels.gml")
    os.remove("Land_Registry_Cadastral_Parcels.gml")
    parcels.to_file("data/parcels.gpkg")
    parcels.to_file("data/parcels.geojson", driver="GeoJSON")
    parcels.to_parquet("data/parcels.parquet")
if not os.path.exists("data/wards.gpkg"):
    reply = requests.get("https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/"
                         "Wards_December_2024_Boundaries_UK_BGC/FeatureServer/0/query",
                         params={"where": "LAD24NM='Birmingham'", "outFields": "WD24CD,WD24NM",
                                 "outSR": 27700, "f": "geojson"}, timeout=120)
    reply.raise_for_status()
    gpd.read_file(io.BytesIO(reply.content)).to_file("data/wards.gpkg")
print(len(gpd.read_file("data/wards.gpkg")), "wards;", os.cpu_count(), "CPUs, load average",
      round(os.getloadavg()[0], 1))

69 wards; 24 CPUs, load average 0.6

Measure peak memory, one strategy per process

Peak memory is a property of a whole process, so each strategy runs in a fresh Python and reports its own peak resident memory (ru_maxrss) and time. The same helper can also run a strategy under an address-space limit (RLIMIT_AS), which is the most direct way to see which of them would survive on a smaller machine.

import resource
import subprocess
import sys
import textwrap

PRE = ("import resource, time, geopandas as gpd\n"
       "start = time.perf_counter()\n")
POST = "\nprint(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, time.perf_counter() - start)"


def peak(code, cap_gb=None):
    limit = None
    if cap_gb:
        limit = lambda: resource.setrlimit(resource.RLIMIT_AS, (int(cap_gb * 2**30),) * 2)
    run = subprocess.run([sys.executable, "-c", PRE + textwrap.dedent(code) + POST],
                         capture_output=True, text=True, preexec_fn=limit)
    if run.returncode:
        last = (run.stderr.strip().splitlines() or ["killed"])[-1]
        return f"failed — {last.strip()[:70]}"
    mb, seconds = map(float, run.stdout.split()[-2:])
    return f"peak {mb:6,.0f} MB  {seconds:5.1f} s"


P = "p = gpd.read_parquet('data/parcels.parquet', columns=['INSPIREID', 'geometry'])\n"
W = "w = gpd.read_file('data/wards.gpkg')\n"
strategies = {
    "import geopandas only":     "pass",
    "read GeoJSON, all":         "g = gpd.read_file('data/parcels.geojson')",
    "read GPKG, all":            "g = gpd.read_file('data/parcels.gpkg')",
    "read GPKG, columns=":       "g = gpd.read_file('data/parcels.gpkg', columns=['INSPIREID'])",
    "read GPKG, bbox= 2 km":     "g = gpd.read_file('data/parcels.gpkg', bbox=(405000, 285500, 407000, 287500))",
    "read Parquet, all":         "g = gpd.read_parquet('data/parcels.parquet')",
    "read Parquet, columns=":    P,
    "overlay with wards":        P + W + "r = p.overlay(w, how='intersection')",
    "sjoin with wards":          P + W + "r = gpd.sjoin(p, w, predicate='intersects')",
    "to_crs, everything":        "p = gpd.read_parquet('data/parcels.parquet'); p = p.to_crs(3857)",
    "filter, then to_crs":       "p = gpd.read_parquet('data/parcels.parquet'); c = p.cx[405000:407000, 285500:287500].to_crs(3857)",
    "three steps, new names":    P + "a = p.to_crs(3857); b = a[a.area > 100]; c = b.to_crs(27700)",
    "three steps, one name":     P + "p = p.to_crs(3857); p = p[p.area > 100]; p = p.to_crs(27700)",
}
for label, code in strategies.items():
    print(f"{label:26} {peak(code)}", flush=True)

import geopandas only      peak    182 MB    0.0 s
read GeoJSON, all          peak    694 MB    5.5 s
read GPKG, all             peak    699 MB    1.2 s
read GPKG, columns=        peak    530 MB    0.5 s
read GPKG, bbox= 2 km      peak    182 MB    0.0 s
read Parquet, all          peak    833 MB    0.4 s
read Parquet, columns=     peak    791 MB    0.4 s
overlay with wards         peak  1,043 MB    5.5 s
sjoin with wards           peak    791 MB    1.3 s
to_crs, everything         peak  1,294 MB    2.9 s
filter, then to_crs        peak    875 MB    0.5 s
three steps, new names     peak  1,451 MB    5.0 s
three steps, one name      peak  1,220 MB    4.9 s

What a memory limit does to each read

for cap in (1.5, 2.0):
    for label in ["read GeoJSON, all", "read GPKG, all", "read GPKG, columns=", "read GPKG, bbox= 2 km"]:
        print(f"{cap} GB limit, {label:22} {peak(strategies[label], cap)}")

1.5 GB limit, read GeoJSON, all      failed — killed
1.5 GB limit, read GPKG, all         failed — MemoryError
1.5 GB limit, read GPKG, columns=    failed — killed
1.5 GB limit, read GPKG, bbox= 2 km  peak    170 MB    0.0 s
2.0 GB limit, read GeoJSON, all      failed — shapely.errors.GEOSException: std::bad_alloc
2.0 GB limit, read GPKG, all         failed — shapely.errors.GEOSException: std::bad_alloc
2.0 GB limit, read GPKG, columns=    peak    529 MB    0.5 s
2.0 GB limit, read GPKG, bbox= 2 km  peak    173 MB    0.0 s

The address-space limit counts memory the process reserves as well as what it uses, so it bites well above the resident peaks in the table before — but it bites the widest reads first. At 2 GB both full reads fail and the narrow ones pass; at 1.5 GB only the bounding-box read survives. Note how the failures are reported: a GEOS exception, a process killed without a message, sometimes a MemoryError — and which one appears varies from run to run. A log searched only for MemoryError misses most of them.

Measured on this page. A narrower read is the difference between failing and finishing.

Batching by bounding box reads the edges twice

wards = gpd.read_file("data/wards.gpkg")
read = sum(len(gpd.read_file("data/parcels.gpkg", bbox=tuple(ward.total_bounds), columns=["INSPIREID"]))
           for _, ward in wards.groupby("WD24CD"))
total = pyogrio.read_info("data/parcels.gpkg")["features"]
print(f"{len(wards)} ward bounding boxes read {read:,} parcels; the file has {total:,} ({read / total:.1f}×)")

69 ward bounding boxes read 650,737 parcels; the file has 338,611 (1.9×)

Write intermediate subsets to disk

from pathlib import Path

Path("temp").mkdir(exist_ok=True)
parcels = gpd.read_parquet("data/parcels.parquet", columns=["INSPIREID", "geometry"])
large = parcels[parcels.area > 10_000]
large.to_parquet("temp/large_parcels.parquet")
print(f"{len(large):,} parcels over 1 ha written; {Path('temp/large_parcels.parquet').stat().st_size / 1e6:.1f} MB")
del large

1,923 parcels over 1 ha written; 5.0 MB

Does `del` give the memory back?

script = textwrap.dedent("""
    import gc, os, geopandas as gpd
    def rss_mb():
        return int(open("/proc/self/statm").read().split()[1]) * os.sysconf("SC_PAGE_SIZE") / 2**20
    before = rss_mb()
    parcels = gpd.read_file("data/parcels.gpkg")
    loaded = rss_mb()
    del parcels
    deleted = rss_mb()
    gc.collect()
    print(f"resident memory: {before:,.0f} MB before, {loaded:,.0f} MB loaded, "
          f"{deleted:,.0f} MB after del, {rss_mb():,.0f} MB after gc.collect()")
""")
print(subprocess.run([sys.executable, "-c", script], capture_output=True, text=True).stdout)

resident memory: 126 MB before, 524 MB loaded, 506 MB after del, 505 MB after gc.collect()

In-memory size bears little relation to file size

GeoPandas stores geometry as in-memory objects, and what they take bears little relation to the file size on disk: the same 338,611 parcels are 62.7 MB as Parquet and 400.9 MB as GeoJSON, and load into about 0.7 GB either way. Text-based formats such as GeoJSON are expensive mainly in time — every coordinate is parsed from text, which made the GeoJSON read about five times slower than the GeoPackage. Heavy operations such as overlay(), buffer(), and dissolve() also create temporary geometry objects, which increases peak memory use.

Measured on this page. The fix is almost always at the read call, not in the processing that follows it.

So the order of fixes follows the measurements: narrow the read first (columns, then area), keep heavy steps on the smallest layer that answers the question (filter before to_crs(), sjoin() where a yes/no relationship is enough), and only then think about format. A faster format pays off on every rerun, but on this data it never lowered the peak.

bbox filtering and file size as a poor predictor

Spatial filtering is not always exact or always available

bbox= filtering depends on the driver: a GeoPackage answers it from its spatial index, while a format without one still parses every feature to test it. And a bounding box returns every feature that touches the rectangle — batching Birmingham by its 69 ward rectangles read 650,737 parcels from a file of 338,611.

File size on disk predicts very little

A 1 GB file does not need 1 GB of RAM, and a 60 MB one can need far more than 60 MB. The parcels needed about 0.7 GB from 400.9 MB of GeoJSON and about 0.8 GB from 62.7 MB of Parquet. Check pyogrio.read_info() for the feature count and field list instead: those are what the read will build.

If the data is valid and the narrow read still does not fit, redesign the workflow around subsets — and filter each subset to its exact area, because bounding-box batches overlap.

Internal links

FAQ

Why does GeoPandas use much more memory than the file size on disk?

Because the file is parsed into in-memory geometry objects and attribute data, and compact formats compress what memory cannot: 62.7 MB of Parquet became about 0.8 GB at peak. Heavy operations then create temporary objects on top — an overlay with the wards pushed the same data past 1 GB.

What file format is best when GeoPandas runs out of memory?

None of them, on its own. The parcels peaked at 694 MB from GeoJSON, 699 MB from GeoPackage and 833 MB from Parquet. Format decides speed — 5.5 s, 1.2 s and 0.4 s for the same reads — while the read's columns= and bbox= decide memory. On a GeoPackage, bbox= uses the file's spatial index, so the rows outside the area are never parsed.

How can I process a large shapefile in GeoPandas without loading everything at once?

Use spatial filtering where supported, split the work by tile or region, and write intermediate outputs to disk instead of keeping the whole workflow in memory.

Does simplifying geometry help fix a GeoPandas memory error?

Yes, when geometry complexity is part of the problem. Simplifying polygons or lines reduces vertex count and can make overlay, buffering, and export lighter.

How do I read only a subset of rows from a large file without loading it all?

If the driver supports it, pass rows= to gpd.read_file() to limit the number of features read, or combine it with bbox= to read only a study area. For Parquet, you can read selected columns with gpd.read_parquet(path, columns=[...]) to lower memory before any processing.

Does deleting a GeoDataFrame with `del` actually free memory?

It frees the objects once nothing else points to them, but the process usually keeps the memory. Measured in a fresh process, reading the parcels took resident memory from 126 MB to 524 MB; after del it was 506 MB, and after gc.collect() 505 MB. The freed memory is reused by the same process, so the next step can fit, but the operating system does not get it back. If a step must hand memory back, run it in a subprocess, and avoid keeping old intermediates bound to other variables.

Why does `to_crs()` cause a memory spike on a large layer?

Reprojection builds a new geometry object for every feature, so peak memory briefly holds both the old and new geometries at once. Filter to the features you actually need first, then call to_crs() on the smaller subset to keep the spike down.

Can I process a large file in row chunks like pandas `chunksize`?

gpd.read_file() has no chunksize, but you can emulate it by reading slices with rows=slice(start, stop) on supported drivers, or by iterating over tiles with bbox= and writing each result to Parquet. For very large workflows, splitting by region or attribute group is usually more reliable than fixed row chunks.