OSError: Too Many Open Files in a Batch GIS Job

Problem statement

The job runs fine for two hours, then falls over on file 4,102 of 9,000.

OSError: [Errno 24] Too many open files: 'data/tiles/tile_04102.gpkg'

rasterio.errors.RasterioIOError: Too many open files
sqlite3.OperationalError: unable to open database file
fiona.errors.DriverError: data/x.shp: No such file or directory      ← misleading

Nothing is wrong with file 4,102. It is the first one the process could not open, because every file opened before it is still open.

$ ls /proc/$(pgrep -f pipeline)/fd | wc -l
1024
$ ulimit -n
1024

Every process has a cap on simultaneously open file descriptors. A batch job that opens files without closing them reaches it after a predictable number of iterations — which is why this always appears partway through a long run and never during testing on ten files.

Quick answer

Find the leak, close the handles, and raise the limit only as a last resort:

# 1. see the leak
import os, psutil
proc = psutil.Process()
print(f"{proc.num_fds()} open descriptors, limit {os.sysconf('SC_OPEN_MAX')}")

# 2. the usual cause — a handle opened and never closed
src = fiona.open(path)          # ← leaks
...

with fiona.open(path) as src:   # ← closes, even on an exception
    ...

# 3. the other usual cause — GeoPandas keeping a reference alive
gdfs = [gpd.read_file(p) for p in paths]     # 9,000 frames, all resident

Symptom	Cause	Fix
fails partway, always around the same count	handles never closed	`with` blocks everywhere
`unable to open database file`	one GeoPackage connection per read	close, or reuse one connection
fails only when parallel	limit is per process, workers multiply the pressure	fewer workers, or close sooner
`No such file or directory` on a file that exists	descriptor exhaustion misreported by the driver	check `num_fds()`
grows steadily even with `with`	matplotlib figures, sockets, or a DB pool	`plt.close()`, dispose the engine

# the diagnostic that names the culprit
import psutil, collections
proc = psutil.Process()
print(collections.Counter(
    (f.path.rsplit(".", 1)[-1] if "." in f.path else "other") for f in proc.open_files()
).most_common(10))
# [('gpkg', 4098), ('tif', 3), ('so', 41), ('other', 12)]

Four thousand GeoPackages open at once names the leak precisely.

Where descriptors go

Grid of common file-descriptor leak sources in a GIS batch job. — Files are the obvious one. Figures, engines and sockets all consume the same budget.

Step-by-step solution

Triage rows pairing each descriptor-exhaustion symptom with its cause and fix. — Raising the limit is the last row for a reason — it postpones the failure rather than removing it.

1. Measure before guessing

import psutil, os

def fd_report():
    proc = psutil.Process()
    soft, hard = __import__("resource").getrlimit(__import__("resource").RLIMIT_NOFILE)
    files = proc.open_files()
    return {
        "open": proc.num_fds(),
        "files": len(files),
        "soft_limit": soft,
        "hard_limit": hard,
        "headroom": soft - proc.num_fds(),
    }

print(fd_report())
# {'open': 1021, 'files': 998, 'soft_limit': 1024, 'hard_limit': 1048576, 'headroom': 3}

Log it every N items and the leak becomes obvious as a rising line rather than a sudden crash:

for i, item in enumerate(work, 1):
    process(item)
    if i % 100 == 0:
        r = fd_report()
        print(f"  {i:>6,} items · {r['open']:>5} fds · {r['headroom']:>5} left")

     100 items ·   112 fds ·   912 left
     200 items ·   212 fds ·   812 left      ← +1 per item: a leak
     300 items ·   312 fds ·   712 left

A flat line is healthy. A line rising by one per item tells you the leak is one handle per iteration, which is almost always a missing with.

2. Close what you open — including the things that do not look like files

# fiona / rasterio: the handle is the resource
with fiona.open(path) as src:
    schema = src.schema
    bounds = src.bounds

with rasterio.open(path) as src:
    band = src.read(1)

# GeoPandas closes its own handle on read_file — but keeps the DATA
gdf = gpd.read_file(path)        # handle closed; memory is the concern instead

gpd.read_file is not usually the leak, because it opens, reads and closes. The leaks are the lower-level handles people open for cheap metadata:

# leaks one descriptor per file
counts = {p: len(fiona.open(p)) for p in paths}

# does not
def count_features(path):
    with fiona.open(path) as src:
        return len(src)
counts = {p: count_features(p) for p in paths}

That first line appears in a lot of inventory scripts, and it is the single most common cause of this error in GIS code.

3. Watch for the non-file descriptors

Matplotlib is the classic surprise:

for tile in tiles:
    fig, ax = plt.subplots()
    tile.plot(ax=ax)
    fig.savefig(out / f"{tile.id}.png")
    # figure never closed — holds memory and, with some backends, descriptors

# fix
    plt.close(fig)

# or
with plt.ioff():
    fig, ax = plt.subplots()
    ...
    plt.close(fig)

Database engines and sockets count too:

# leaks a connection pool per call
def enrich(path):
    engine = create_engine(URL)        # ← new pool, every file
    ...

# one engine per process
@lru_cache(maxsize=1)
def get_engine():
    return create_engine(URL, pool_size=5, max_overflow=5)

A SQLAlchemy engine holds up to pool_size + max_overflow sockets. Nine hundred engines is nine hundred pools.

4. Do not keep frames you have finished with

# holds every frame for the whole run
frames = []
for path in paths:
    frames.append(gpd.read_file(path))
combined = pd.concat(frames)

# streams, keeping one frame at a time
def stream(paths):
    for path in paths:
        yield gpd.read_file(path)
combined = pd.concat(stream(paths), ignore_index=True)

This is a memory problem more than a descriptor one, but it produces the same crash on some systems because memory-mapped readers hold a descriptor for as long as the array is alive. See batch GIS job slows down and runs out of memory.

5. Understand how parallelism multiplies the pressure

The limit is per process, so eight workers each opening 200 files is fine. What is not fine:

# the parent holds a handle per submitted task's result
futures = {pool.submit(process, p): p for p in 9000_paths}
for fut in as_completed(futures):
    ...

Each worker also inherits the parent's open descriptors at fork time. A parent that already holds 900 handles gives every child 900 to start with:

# close before forking
def main():
    inventory = build_inventory(paths)        # opens and closes cleanly
    gc.collect()                              # drop lingering handles
    with ProcessPoolExecutor(8) as pool:      # children start clean
        ...

And use maxtasksperchild so a slow leak in a worker is bounded:

import multiprocessing as mp
with mp.Pool(8, maxtasksperchild=200) as pool:      # worker restarts every 200 items
    pool.map(process, sorted(items))

That is a mitigation, not a fix — but it turns a run that dies at item 4,102 into one that finishes while you find the real leak.

6. Raise the limit last, and deliberately

ulimit -n              # 1024 soft
ulimit -Hn             # 1048576 hard
ulimit -n 8192         # raise the soft limit for this shell

import resource
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_NOFILE, (min(8192, hard), hard))

# systemd unit — the shell ulimit does not apply to a service
[Service]
LimitNOFILE=8192

Raising the limit is legitimate when the job genuinely needs many handles at once — writing a thousand output tiles concurrently, say. It is illegitimate as a response to a leak, because a leak grows without bound and 8,192 only postpones the crash to item 32,000.

The honest test: does the descriptor count plateau? If it rises linearly with items processed, no limit is high enough.

Code examples

Example 1: a leak detector you can leave switched on

import psutil, resource
from contextlib import contextmanager

class FDWatch:
    def __init__(self, warn_at=0.8):
        self.proc = psutil.Process()
        self.soft, self.hard = resource.getrlimit(resource.RLIMIT_NOFILE)
        self.warn_at = warn_at
        self.baseline = self.proc.num_fds()
        self.peak = self.baseline

    def check(self, label=""):
        n = self.proc.num_fds()
        self.peak = max(self.peak, n)
        if n > self.soft * self.warn_at:
            top = collections.Counter(
                f.path.rsplit(".", 1)[-1] for f in self.proc.open_files() if "." in f.path
            ).most_common(5)
            raise RuntimeError(
                f"{label}: {n}/{self.soft} descriptors open (peak {self.peak}). "
                f"Most common: {top}"
            )
        return n

    @contextmanager
    def around(self, label):
        before = self.proc.num_fds()
        yield
        after = self.proc.num_fds()
        if after > before:
            print(f"  ⚠ {label} leaked {after - before} descriptor(s)")

watch = FDWatch()
for i, item in enumerate(work, 1):
    with watch.around(f"item {i}"):
        process(item)
    if i % 50 == 0:
        watch.check(f"after {i} items")

The around context manager is the useful half — it names the step that leaks, not just the fact that something does. Wrap a few candidate steps and the culprit identifies itself on the first iteration.

Example 2: a safe inventory that does not leak

import fiona
from pathlib import Path

def inventory(paths):
    """Metadata for many files without holding a single handle open."""
    rows = []
    for path in paths:
        try:
            with fiona.open(path) as src:
                rows.append({
                    "path": str(path), "layer": src.name,
                    "features": len(src), "crs": str(src.crs),
                    "geometry": src.schema.get("geometry"),
                    "bounds": src.bounds,
                })
        except Exception as exc:
            rows.append({"path": str(path), "error": f"{type(exc).__name__}: {exc}"})
    return rows

Compare with the version that causes the error:

handles = {p: fiona.open(p) for p in paths}     # every handle open, forever
rows = [{"path": p, "features": len(h)} for p, h in handles.items()]

The second reads more naturally, which is exactly why it gets written. See how to build an inventory of a GIS data folder.

Example 3: a test that catches the leak before production does

import psutil

def test_processing_does_not_leak_descriptors(tmp_path, many_small_files):
    proc = psutil.Process()
    process_one(many_small_files[0])          # warm up: imports, caches, drivers
    baseline = proc.num_fds()

    for path in many_small_files[:50]:
        process_one(path)

    leaked = proc.num_fds() - baseline
    assert leaked <= 2, f"leaked {leaked} descriptors over 50 items"

The warm-up call matters: the first invocation legitimately opens shared libraries, PROJ's database and driver caches. Measuring from the second call onward isolates the per-item behaviour, which is the thing that must be flat.

Fifty items is enough — a one-per-item leak shows as 50, and no threshold hides it.

Explanation

Two panels contrasting a flat descriptor count with one rising linearly per item. — Flat means correct. Rising means no limit is high enough.

A file descriptor is a small integer the kernel gives a process to refer to an open file, socket, pipe or device. Each process has a soft limit (commonly 1,024) and a hard limit (often over a million), and the soft limit is what produces Errno 24.

GIS work consumes descriptors faster than most Python because of what a "file" is in this domain. A shapefile is four to six actual files — .shp, .shx, .dbf, .prj, sometimes .cpg and .qix — and GDAL may hold several of them at once. A GeoPackage is a SQLite database, so opening it takes a connection which may itself use more than one descriptor. Reading a raster with overviews can open sidecar files. A loop over 300 shapefiles that leaks one handle each is really leaking closer to 1,500.

That multiplier explains why the failure appears at an unpredictable count and why it is worse for shapefiles than GeoPackages.

The error message is frequently misleading. When a driver cannot obtain a descriptor, some report the honest Too many open files and others report No such file or directory for a file that plainly exists, because the underlying open() failed and the error was flattened. Checking num_fds() at the moment of failure resolves the ambiguity in one line — and is worth doing before spending an hour on a path bug that does not exist.

The distinction that matters operationally is plateau versus climb. A job that legitimately needs many concurrent handles — writing 500 tiles in parallel — has a count that rises and then stays flat. A leak rises linearly with items processed and never comes down. Raising ulimit is a reasonable answer to the first and a way of hiding the second: at one leaked handle per item and a limit of 8,192, the job now dies at item 8,000 instead of item 1,000, after four times as long.

Edge cases or notes

macOS defaults to a much lower limit than Linux — often 256. A job that works on Linux may fail immediately on a Mac.
ulimit in a shell does not affect a systemd service. Set LimitNOFILE= in the unit file.
Docker containers inherit the daemon's limits; --ulimit nofile=8192:8192 on docker run.
psutil.Process().open_files() does not list sockets — use connections() for those.
A shapefile is 4–6 descriptors, not one. Divide the count by that when estimating capacity.
GDAL has its own file cache (GDAL_MAX_DATASET_POOL_SIZE) that keeps datasets open deliberately; lowering it can help.
gc.collect() closes handles held by unreferenced objects — useful as a diagnostic, not as a fix.
maxtasksperchild is multiprocessing.Pool only; ProcessPoolExecutor gained max_tasks_per_child in Python 3.11.
Reading with pyogrio opens and closes per call, so it leaks less than a held fiona handle — another reason to prefer it.

Internal links

Batch GIS job slows down and runs out of memory — the sibling resource-exhaustion failure
Parallel GIS batch job hangs or crashes with multiprocessing — where worker counts multiply pressure
How to build an inventory of a GIS data folder in Python — the script that most often causes this
How to speed up batch GIS jobs with parallel processing — maxtasksperchild and worker lifetime
The anatomy of a batch job: discover, apply, report — why apply_one should not hold shared handles
Fiona vs pyogrio: how GeoPandas reads and writes files — which engine holds what
How to process a very large GeoPackage in chunks — one connection, many reads
GeoPandas cannot connect to PostGIS — engine-per-call as a connection leak

FAQ

Why does it always fail around the same file number?

Because the limit is a fixed count and the leak is a fixed rate. One handle per item against a 1,024 limit fails at roughly item 1,000, every time.

Is `gpd.read_file` leaking?

Rarely — it opens, reads and closes. The leaks are usually lower-level handles kept for metadata, like fiona.open(p) in a dict comprehension.

Should I just raise ulimit?

Only if the descriptor count plateaus. If it climbs linearly with items processed, raising the limit postpones the crash rather than fixing it.

Why does it only fail when I run in parallel?

The limit is per process, but children inherit the parent's open descriptors at fork, and the parent may hold handles for results. Close before forking and cap worker lifetime.

Why does the error say "No such file or directory"?

Some drivers flatten a failed open() into a path error. Check psutil.Process().num_fds() at the point of failure to tell the two apart.

Does matplotlib really leak descriptors?

It leaks figures, which hold memory and — with some backends — descriptors. plt.close(fig) after every save is the fix.

How do I test for this?

Warm up once, record num_fds(), process fifty items, and assert the count grew by no more than one or two.

OSError: Too Many Open Files in a Batch GIS Job #

Problem statement #

Quick answer #

Where descriptors go #

Step-by-step solution #

1. Measure before guessing #

2. Close what you open — including the things that do not look like files #

3. Watch for the non-file descriptors #

4. Do not keep frames you have finished with #

5. Understand how parallelism multiplies the pressure #

6. Raise the limit last, and deliberately #

Code examples #

Example 1: a leak detector you can leave switched on #

Example 2: a safe inventory that does not leak #

Example 3: a test that catches the leak before production does #

Explanation #

Edge cases or notes #

Internal links #

FAQ #

Why does it always fail around the same file number? #

Is gpd.read_file leaking? #

Should I just raise ulimit? #

Why does it only fail when I run in parallel? #

Why does the error say "No such file or directory"? #

Does matplotlib really leak descriptors? #

How do I test for this? #