OSError: Too Many Open Files in a Batch GIS Job

Problem statement

The job runs fine for a while, then falls over partway through a folder. Here it is in miniature — one GeoPackage per country, 258 of them, and a process allowed 128 open files, as a tight service limit might allow (the setup is under The data used on this page):

import gc
import resource
from pathlib import Path

import fiona

paths = sorted(Path("data/tiles").glob("*.gpkg"))
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print(f"this process may open {soft:,} files; lowering that to 128 for the demonstration")
resource.setrlimit(resource.RLIMIT_NOFILE, (128, hard))

try:
    counts = {p: len(fiona.open(p)) for p in paths}          # looks harmless
except Exception as err:
    print(f"{type(err).__name__}: {err}")

    print(f"  caused by {type(err.__cause__).__name__}: {err.__cause__}")

# a little headroom, so the diagnostics below can run while the leak is still live
resource.setrlimit(resource.RLIMIT_NOFILE, (256, hard))

this process may open 524,288 files; lowering that to 128 for the demonstration
DriverError: Failed to open dataset (flags=68): data/tiles/KAB.gpkg
  caused by CPLE_OpenFailedError: data/tiles/KAB.gpkg: Too many open files

Nothing is wrong with the file named in the error. It is the first one the process could not open, because every file opened before it is still open. And notice what the top-level exception says: failed to open dataset — the words "too many open files" are only in the chained cause, which a log that records str(err) never shows. That is why this failure so often gets investigated as a corrupt file.

Every process has a cap on simultaneously open file descriptors. A batch job that opens files without closing them reaches it after a predictable number of iterations — which is why this always appears partway through a long run and never during testing on ten files.

Quick answer

Find the leak, close the handles, and raise the limit only as a last resort:

# 1. see the leak
import os, psutil
proc = psutil.Process()
print(f"{proc.num_fds()} open descriptors, limit {os.sysconf('SC_OPEN_MAX')}")

# 2. the usual cause — a handle opened and never closed
src = fiona.open(path)          # ← leaks
...

with fiona.open(path) as src:   # ← closes, even on an exception
    ...

# 3. the other usual cause — GeoPandas keeping a reference alive
gdfs = [gpd.read_file(p) for p in paths]     # 9,000 frames, all resident

# the diagnostic that names the culprit — run while the leak above is still live
import collections

import psutil

proc = psutil.Process()
print(collections.Counter(
    (f.path.rsplit(".", 1)[-1] if "." in f.path else "other") for f in proc.open_files()
).most_common(5))

[('gpkg', 119), ('db', 3), ('log', 2), ('0', 1)]

A hundred and nineteen GeoPackages open at once names the leak precisely. (open_files lists regular files only; the db and log entries are the notebook's own history files, and the rest of the descriptors are pipes and sockets.)

Where descriptors go

Measure, close everything, drop finished frames

1. Measure before guessing

import psutil, os

def fd_report():
    proc = psutil.Process()
    soft, hard = __import__("resource").getrlimit(__import__("resource").RLIMIT_NOFILE)
    files = proc.open_files()
    return {
        "open": proc.num_fds(),
        "files": len(files),
        "soft_limit": soft,
        "hard_limit": hard,
        "headroom": soft - proc.num_fds(),
    }

print(fd_report())

{'open': 129, 'files': 125, 'soft_limit': 256, 'hard_limit': 524288, 'headroom': 127}

Log it every N items and the leak becomes obvious as a rising line rather than a sudden crash:

gc.collect()                                     # release the handles leaked above


def process(path):                               # the same leak, as a per-item step
    return len(fiona.open(path))


work = paths[:90]
for i, item in enumerate(work, 1):
    process(item)
    if i % 30 == 0:
        r = fd_report()
        print(f"  {i:>6,} items · {r['open']:>5} fds · {r['headroom']:>5} left")
gc.collect()
print(f"after gc.collect(): {fd_report()['open']} fds")

      30 items ·    40 fds ·   216 left
      60 items ·    70 fds ·   186 left
      90 items ·   100 fds ·   156 left
after gc.collect(): 10 fds

A flat line is healthy. A line rising by one per item tells you the leak is one handle per iteration, which is almost always a missing with. The last line is the subtle part: these handles are not lost for ever. They sit in reference cycles, and gc.collect() closes every one of them — but the garbage collector runs when enough objects have been allocated, not when descriptors run low, so a job can exhaust its limit long before the collector gets round to it. Closing explicitly is the only thing that makes the count predictable.

2. Close what you open — including the things that do not look like files

# fiona / rasterio: the handle is the resource
with fiona.open(path) as src:
    schema = src.schema
    bounds = src.bounds

with rasterio.open(path) as src:
    band = src.read(1)

# GeoPandas closes its own handle on read_file — but keeps the DATA
gdf = gpd.read_file(path)        # handle closed; memory is the concern instead

gpd.read_file is not usually the leak, because it opens, reads and closes. The leaks are the lower-level handles people open for cheap metadata:

# leaks one descriptor per file
counts = {p: len(fiona.open(p)) for p in paths}

# does not
def count_features(path):
    with fiona.open(path) as src:
        return len(src)
counts = {p: count_features(p) for p in paths}

That first line appears in a lot of inventory scripts, and it is the single most common cause of this error in GIS code.

3. Watch for the non-file descriptors

Matplotlib is the classic surprise:

for tile in tiles:
    fig, ax = plt.subplots()
    tile.plot(ax=ax)

    fig.savefig(out / f"{tile.id}.png")
    # figure never closed — pyplot keeps every one alive (it warns at 20 open)

# fix
    plt.close(fig)

# or
with plt.ioff():
    fig, ax = plt.subplots()
    ...
    plt.close(fig)

Database engines and sockets count too:

# leaks a connection pool per call
def enrich(path):
    engine = create_engine(URL)        # ← new pool, every file
    ...

# one engine per process
@lru_cache(maxsize=1)
def get_engine():
    return create_engine(URL, pool_size=5, max_overflow=5)

A SQLAlchemy engine holds up to pool_size + max_overflow sockets. Nine hundred engines is nine hundred pools.

4. Do not keep frames you have finished with

# holds every frame for the whole run
frames = []
for path in paths:
    frames.append(gpd.read_file(path))
combined = pd.concat(frames)

# streams, keeping one frame at a time
def stream(paths):
    for path in paths:
        yield gpd.read_file(path)
combined = pd.concat(stream(paths), ignore_index=True)

This is a memory problem more than a descriptor one, but it produces the same crash on some systems because memory-mapped readers hold a descriptor for as long as the array is alive. See batch GIS job slows down and runs out of memory.

5. Understand how parallelism multiplies the pressure

The limit is per process, so eight workers each opening 200 files is fine, and a leak inside a worker only exhausts that worker. What crosses the process boundary is the parent's state: with the fork start method each child inherits the parent's open descriptors, so a parent that already holds 900 handles gives every child 900 to start with. (Python 3.14 made forkserver the default on Linux, which starts children clean; code that asks for fork explicitly, or runs on an older Python, still inherits.) Measured, with the parent holding 200 GeoPackages open:

%%writefile fdprobe.py
import psutil


def open_fds(_=None):
    return psutil.Process().num_fds()

import multiprocessing as mp

from fdprobe import open_fds

held = [fiona.open(p) for p in paths[:200]]          # a parent that leaked
print(f"default start method: {mp.get_start_method()}; parent holds {open_fds()} fds")
for method in ("fork", "forkserver", "spawn"):
    with mp.get_context(method).Pool(1) as pool:
        print(f"  {method:<10} child starts with {pool.apply(open_fds):>3} fds")
for h in held:
    h.close()

default start method: forkserver; parent holds 212 fds
  fork       child starts with 218 fds
  forkserver child starts with  12 fds
  spawn      child starts with  11 fds

A forked child starts with the parent's 200 GeoPackages already open and no way to close them — it does not even know they are there.

# close before forking
def main():
    inventory = build_inventory(paths)        # opens and closes cleanly
    gc.collect()                              # drop lingering handles
    with ProcessPoolExecutor(8) as pool:      # children start clean
        ...

And use maxtasksperchild so a slow leak in a worker is bounded:

import multiprocessing as mp
with mp.Pool(8, maxtasksperchild=200) as pool:      # worker restarts every 200 items
    pool.map(process, sorted(items))

That is a mitigation, not a fix — but it turns a run that dies at item 4,102 into one that finishes while you find the real leak.

6. Raise the limit last, and deliberately

ulimit -n              # the soft limit — 1,024 is the classic default
ulimit -Hn             # the hard limit, the ceiling an unprivileged process can raise to
ulimit -n 8192         # raise the soft limit for this shell

import resource
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_NOFILE, (min(8192, hard), hard))
print(f"soft limit {soft:,} → {resource.getrlimit(resource.RLIMIT_NOFILE)[0]:,} (hard {hard:,})")

soft limit 256 → 8,192 (hard 524,288)

# systemd unit — the shell ulimit does not apply to a service
[Service]
LimitNOFILE=8192

Raising the limit is legitimate when the job genuinely needs many handles at once — writing a thousand output tiles concurrently, say. It is illegitimate as a response to a leak, because a leak grows without bound and 8,192 only postpones the crash to around item 8,000 — or 4,000 for shapefiles, at two descriptors a handle.

The honest test: does the descriptor count plateau? If it rises linearly with items processed, no limit is high enough.

A leak detector and a test that catches leaks

The data used on this page

Natural Earth's 1:10m countries, one GeoPackage per country — 258 small real files in data/tiles/, enough to exhaust a descriptor limit lowered to 128 for the demonstrations. The page lowers the limit with resource.setrlimit in step 1 and raises it again in step 6, so nothing it does outlasts the page.

import sys
from pathlib import Path

import geopandas as gpd

world = gpd.read_file("https://naciscdn.org/naturalearth/10m/cultural/ne_10m_admin_0_countries.zip",
                      columns=["ADM0_A3", "NAME"])
Path("data/tiles").mkdir(parents=True, exist_ok=True)
for code in world["ADM0_A3"]:
    world[world["ADM0_A3"] == code].to_file(f"data/tiles/{code}.gpkg")
print(f"{len(list(Path('data/tiles').glob('*.gpkg')))} files")

258 files

Example 1: a leak detector you can leave switched on

import collections
import resource
from contextlib import contextmanager

import psutil


class FDWatch:
    def __init__(self, warn_at=0.8):
        self.proc = psutil.Process()
        self.soft, self.hard = resource.getrlimit(resource.RLIMIT_NOFILE)
        self.warn_at = warn_at
        self.baseline = self.proc.num_fds()
        self.peak = self.baseline

    def check(self, label=""):
        n = self.proc.num_fds()
        self.peak = max(self.peak, n)
        if n > self.soft * self.warn_at:
            top = collections.Counter(
                f.path.rsplit(".", 1)[-1] for f in self.proc.open_files() if "." in f.path
            ).most_common(5)
            raise RuntimeError(
                f"{label}: {n}/{self.soft} descriptors open (peak {self.peak}). "
                f"Most common: {top}"
            )
        return n

    @contextmanager
    def around(self, label):
        before = self.proc.num_fds()
        yield
        after = self.proc.num_fds()
        if after > before:
            print(f"  ⚠ {label} leaked {after - before} descriptor(s)")

watch = FDWatch()
for i, item in enumerate(work[:3], 1):
    with watch.around(f"item {i}"):
        process(item)                     # the leaking step from step 1
    if i % 50 == 0:
        watch.check(f"after {i} items")
gc.collect()

  ⚠ item 1 leaked 1 descriptor(s)
  ⚠ item 2 leaked 1 descriptor(s)
  ⚠ item 3 leaked 1 descriptor(s)

The around context manager is the useful half — it names the step that leaks, not just the fact that something does. Wrap a few candidate steps and the culprit identifies itself on the first iteration.

Example 2: a safe inventory that does not leak

import fiona
from pathlib import Path

def inventory(paths):
    """Metadata for many files without holding a single handle open."""
    rows = []
    for path in paths:
        try:
            with fiona.open(path) as src:
                rows.append({
                    "path": str(path), "layer": src.name,
                    "features": len(src), "crs": str(src.crs),
                    "geometry": src.schema.get("geometry"),
                    "bounds": src.bounds,
                })
        except Exception as exc:
            rows.append({"path": str(path), "error": f"{type(exc).__name__}: {exc}"})
    return rows


before = fd_report()["open"]
rows = inventory(paths)
print(f"{len(rows)} files inventoried; open descriptors {before} before, {fd_report()['open']} after")

258 files inventoried; open descriptors 21 before, 21 after

Compare with the version that causes the error:

handles = {p: fiona.open(p) for p in paths}     # every handle open, forever
rows = [{"path": p, "features": len(h)} for p, h in handles.items()]

The second reads more naturally, which is exactly why it gets written. See how to build an inventory of a GIS data folder.

Example 3: a test that catches the leak before production does

%%writefile test_fds.py
from pathlib import Path

import fiona
import psutil
import pytest


@pytest.fixture
def many_small_files():
    return sorted(Path("data/tiles").glob("*.gpkg"))[:60]


def process_one(path):
    with fiona.open(path) as src:
        return len(src)


def process_one_leaky(path):
    return len(fiona.open(path))


def leaked_over_50(fn, files):
    proc = psutil.Process()
    fn(files[0])                                   # warm up: imports, caches, drivers
    baseline = proc.num_fds()
    for path in files[1:51]:
        fn(path)
    return proc.num_fds() - baseline


def test_processing_does_not_leak_descriptors(many_small_files):
    leaked = leaked_over_50(process_one, many_small_files)
    assert leaked <= 2, f"leaked {leaked} descriptors over 50 items"


@pytest.mark.xfail(strict=True, reason="the leaky version must be caught")
def test_the_leaky_version_is_caught(many_small_files):
    leaked = leaked_over_50(process_one_leaky, many_small_files)
    assert leaked <= 2, f"leaked {leaked} descriptors over 50 items"

!{sys.executable} -m pytest -q --no-header -p no:cacheprovider --color=no -rxX test_fds.py

.x                                                                       [100%]
=========================== short test summary info ============================
XFAIL test_fds.py::test_the_leaky_version_is_caught - the leaky version must be caught
1 passed, 1 xfailed in 0.30s

The warm-up call matters: the first invocation legitimately opens shared libraries, PROJ's database and driver caches. Measuring from the second call onward isolates the per-item behaviour, which is the thing that must be flat.

Fifty items is enough — a one-per-item leak shows as 50, and no threshold hides it.

What a file descriptor is

A file descriptor is a small integer the kernel gives a process to refer to an open file, socket, pipe or device. Each process has a soft limit and a hard limit, and the soft limit is what produces Errno 24. The soft limit varies more than people expect: 1,024 is the classic Linux default and still what a systemd service gets unless its unit says otherwise, while the interactive session this page ran in allowed 524,288.

GIS work consumes descriptors faster than most Python because of what a "file" is in this domain. A shapefile is several actual files — .shp, .shx, .dbf, .prj, sometimes .cpg and .qix — and an open handle holds more than one of them. Measured with fiona on this site's machine, an open shapefile held two descriptors (the .shp and the .dbf; the others are read and closed) and an open GeoPackage held one. Reading a raster with overviews can open sidecar files. A loop over 300 shapefiles that leaks one handle each is really leaking 600 descriptors.

That multiplier explains why the failure appears at a different count for different formats, and sooner for shapefiles than GeoPackages.

The error message is frequently misleading. When a driver cannot obtain a descriptor, the honest Too many open files may be buried: on this page fiona raised DriverError: Failed to open dataset and kept the real reason in the chained exception, and other drivers report No such file or directory for a file that plainly exists, because the underlying open() failed and the error was flattened. Checking num_fds() at the moment of failure resolves the ambiguity in one line — and is worth doing before spending an hour on a path bug that does not exist.

The distinction that matters operationally is plateau versus climb. A job that legitimately needs many concurrent handles — writing 500 tiles in parallel — has a count that rises and then stays flat. A leak rises linearly with items processed and never comes down. Raising ulimit is a reasonable answer to the first and a way of hiding the second: at one leaked handle per item and a limit of 8,192, the job now dies at item 8,000 instead of item 1,000, after eight times as long.

macOS limits, systemd and Docker

• macOS defaults to a much lower limit than Linux — often 256. A job that works on Linux may fail immediately on a Mac.
• ulimit in a shell does not affect a systemd service. Set LimitNOFILE= in the unit file.
• Docker containers inherit the daemon's limits; --ulimit nofile=8192:8192 on docker run.
• psutil.Process().open_files() does not list sockets — use connections() for those.
• A shapefile handle held two descriptors, a GeoPackage one, measured with fiona. Divide by that when estimating capacity.
• GDAL has its own file cache (GDAL_MAX_DATASET_POOL_SIZE) that keeps datasets open deliberately; lowering it can help.
• gc.collect() closes handles held by unreferenced objects — on this page it released 90 leaked handles at once. Useful as a diagnostic, not as a fix: the collector runs on allocations, not on descriptor pressure.
• maxtasksperchild is multiprocessing.Pool only; ProcessPoolExecutor gained max_tasks_per_child in Python 3.11.
• Reading with pyogrio opens and closes per call, so it leaks less than a held fiona handle — another reason to prefer it.

Internal links

• Batch GIS job slows down and runs out of memory — the sibling resource-exhaustion failure
• Parallel GIS batch job hangs or crashes with multiprocessing — where worker counts multiply pressure
• How to build an inventory of a GIS data folder in Python — the script that most often causes this
• How to speed up batch GIS jobs with parallel processing — maxtasksperchild and worker lifetime
• The anatomy of a batch job: discover, apply, report — why apply_one should not hold shared handles
• Fiona vs pyogrio: how GeoPandas reads and writes files — which engine holds what
• How to process a very large GeoPackage in chunks — one connection, many reads
• GeoPandas cannot connect to PostGIS — engine-per-call as a connection leak

FAQ

Why does it always fail around the same file number?

Because the limit is a fixed count and the leak is a fixed rate. One handle per item against a 1,024 limit fails at roughly item 1,000, every time; on this page, with the limit at 128, it was KAB.gpkg, the 120th file, on every run.

Is gpd.read_file leaking?

Rarely — it opens, reads and closes. The leaks are usually lower-level handles kept for metadata, like fiona.open(p) in a dict comprehension.

Should I just raise ulimit?

Only if the descriptor count plateaus. If it climbs linearly with items processed, raising the limit postpones the crash rather than fixing it.

Why does it only fail when I run in parallel?

The limit is per process, so eight workers each have their own budget — but with the fork start method every child also inherits the parent's open descriptors. A parent that leaks before forking hands each worker a head start towards the limit. Close before forking, and cap worker lifetime so a slow leak inside a worker is bounded.

Why does the error say "No such file or directory"?

Some drivers flatten a failed open() into a path error. Check psutil.Process().num_fds() at the point of failure to tell the two apart.

Does matplotlib really leak descriptors?

Mostly it leaks figures: pyplot keeps every figure alive until it is closed, and warns once twenty are open. That is a memory leak first. plt.close(fig) after every save is the fix either way.

How do I test for this?

Warm up once, record num_fds(), process fifty items, and assert the count grew by no more than one or two.