GIS Tests Run Out of Memory or Time Out in CI: How to Fix It

Problem statement

The suite passes locally in a minute or two. In GitHub Actions:

Error: The operation was canceled.
##[error]The job running on runner GitHub Actions 12 has exceeded the maximum
execution time of 360 minutes.

Or it dies without explanation:

##[error]Process completed with exit code 137.

Exit code 137 is 128 + 9 — the process received SIGKILL. Nothing in the logs says why, because the kernel's OOM killer does not ask permission and pytest never gets a chance to report.

Or it simply crawls — many times slower than on a laptop — so nobody waits for it and the checks get merged around.

CI runners are small — GitHub's standard Linux runner has a handful of cores and single-digit to mid-teens gigabytes of RAM depending on the plan, and self-hosted runners in containers are often smaller still — and GIS libraries are written on the assumption that memory is plentiful. Most of the gap between local and CI comes from three or four specific behaviours, all of which are fixable.

Quick answer

env:
  GDAL_CACHEMAX: 256          # MB — the default is a share of HOST memory
  OMP_NUM_THREADS: 1          # stop each library spawning a thread per core
  OPENBLAS_NUM_THREADS: 1
  NUMEXPR_NUM_THREADS: 1
  PROJ_NETWORK: OFF           # no grid downloads mid-test

pytest -m "not slow" -p no:cacheprovider --timeout=60 -x -q

Cap GDAL_CACHEMAX first. It is one line and it is the single most common cause of exit 137.

Exit codes, GDAL_CACHEMAX, threads, downloads, timeouts

1. Read the exit code

exit 137  =  128 + 9   → SIGKILL   → almost always the OOM killer
exit 143  =  128 + 15  → SIGTERM   → cancelled, or a timeout
exit 124                → the `timeout` command fired
exit 1                  → tests genuinely failed

Exit 137 means the kernel killed the process. pytest produced no report because it was not asked. Confirm it by watching memory as the job runs:

- name: Watch memory
  run: |
    ( while true; do
        free -m | awk '/^Mem:/ {printf "%s used=%sMB avail=%sMB\n", strftime("%H:%M:%S"), $3, $7}'
        sleep 5
      done ) &
    echo $! > /tmp/memwatch.pid

- name: Tests
  run: pytest -q

- name: Stop watching
  if: always()
  run: kill "$(cat /tmp/memwatch.pid)" 2>/dev/null || true

On the runner, the log shows available memory falling towards zero in the seconds before the kill, and the step it happened in names the test.

You can reproduce the kill locally, which is much faster than iterating in CI. On Linux, systemd-run puts a command in a memory-capped cgroup, as a container would:

systemd-run --user --scope -q -p MemoryMax=300M -p MemorySwapMax=0 \
  python -c "
import numpy as np
blocks = []
for i in range(20):
    blocks.append(np.ones(50_000_000, dtype=np.uint8))
    print(i, (i + 1) * 50, 'MB', flush=True)
"
echo "exit $?"

0 50 MB
1 100 MB
2 150 MB
3 200 MB
4 250 MB
Killed
exit 137

That is the whole signature: output that simply stops, the shell's Killed, exit 137, and no Python traceback anywhere — the process never got the chance to write one.

2. Cap GDAL_CACHEMAX — the commonest single cause

GDAL's block cache defaults to a percentage of system memory — 5% — and "system" means the host, not the container's limit:

import psutil
import rasterio
from rasterio.env import get_gdal_config

with rasterio.Env():
    cache = int(get_gdal_config("GDAL_CACHEMAX"))
print(f"host memory {psutil.virtual_memory().total / 2**30:.1f} GiB → GDAL cache {cache / 2**30:.2f} GiB")

host memory 60.9 GiB → GDAL cache 3.05 GiB

On this 64 GB machine that is over 3 GiB of cache, and GDAL will fill it happily inside a container limited to 2 GB.

env:
  GDAL_CACHEMAX: 256        # MB, since GDAL 3.x accepts a plain number as MB

# or from Python, around the work that opens rasters
with rasterio.Env(GDAL_CACHEMAX=256):
    print(f"GDAL cache inside the block: {int(get_gdal_config('GDAL_CACHEMAX')):,} MB")

GDAL cache inside the block: 256 MB

Two related settings matter for raster work:

  GDAL_SWATH_SIZE: 33554432          # 32 MB per warp swath, default is larger
  GDAL_DISABLE_READDIR_ON_OPEN: EMPTY_DIR   # do not list a whole directory per open
  VSI_CACHE: FALSE                   # no extra /vsi file cache in a short-lived job

GDAL_DISABLE_READDIR_ON_OPEN is a speed fix rather than a memory one, and a large one on network or object storage: without it, opening one file lists the entire directory.

3. Stop libraries oversubscribing the CPU

NumPy's BLAS, GDAL and others spawn worker threads sized from the CPU count they detect — and how a runner limits CPU decides whether they detect it correctly. Here is one NumPy workload (300 multiplications of a 600 × 600 matrix) on a 24-thread machine, limited to two cores in the two ways Linux offers:

# affinity: the process may only run on cores 0 and 1 (taskset, a cpuset)
taskset -c 0,1 python blas.py                                         # 0.93 s
OPENBLAS_NUM_THREADS=1 taskset -c 0,1 python blas.py                  # 1.90 s

# quota: any core, but two cores' worth of time (docker --cpus=2, a Kubernetes limit)
systemd-run --user --scope -q -p CPUQuota=200% python blas.py         # 8.48 s
OPENBLAS_NUM_THREADS=1 systemd-run ... -p CPUQuota=200% python blas.py  # 1.80 s
OPENBLAS_NUM_THREADS=2 systemd-run ... -p CPUQuota=200% python blas.py  # 0.89 s

Under affinity, OpenBLAS sees the two cores it may use and sizes itself correctly — forcing one thread only halves the speed. Under a quota, the process still sees all 24 CPUs (os.cpu_count() and even sched_getaffinity report 24), OpenBLAS starts 24 threads, and they fight over two cores' worth of time: nearly ten times slower than two threads. Container CPU limits are quotas, which is why this bites in self-hosted runners and Kubernetes pods far more than on a hosted VM.

env:                      # the cores the job really has — 2 here
  OMP_NUM_THREADS: 2
  OPENBLAS_NUM_THREADS: 2
  MKL_NUM_THREADS: 2
  NUMEXPR_NUM_THREADS: 2
  GDAL_NUM_THREADS: 2

# these must be set before the libraries are imported
import os
for var in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ.setdefault(var, "2")
import numpy, geopandas          # now they see the limit

Setting them to 1 is the common advice and a safe default when you do not know the quota; the measurement above shows what it costs when you do.

Setting them in the workflow's env: block is more reliable than in conftest.py, because it applies before Python starts. Thread limits read at import time are ignored if set afterwards.

Each thread also has a stack and its own buffers, so oversubscription costs memory as well as time — which is why this sometimes fixes an exit 137 rather than only a slow job.

4. Do not download anything during tests

Network work in a test is slow, flaky, and occasionally silent:

# ❌ every run, over the network
@pytest.fixture
def uk_boundary():
    return gpd.read_file("https://example.org/uk.geojson")

# ✅ committed, tiny, offline
@pytest.fixture(scope="session")
def uk_boundary():
    return gpd.read_file(Path(__file__).parent / "data" / "uk_simplified.gpkg")

The subtler network dependency is PROJ. With PROJ_NETWORK=ON, a datum transformation may fetch a grid from a CDN mid-test — adding seconds, and making results depend on network state:

env:
  PROJ_NETWORK: OFF

def test_no_network_during_tests(monkeypatch):
    """Fail loudly if anything tries to open a socket."""
    import socket
    def blocked(*args, **kwargs):
        raise RuntimeError("a test attempted a network connection")
    monkeypatch.setattr(socket.socket, "connect", blocked)
    run_the_pipeline_under_test()

Blocking sockets is a blunt instrument and an effective one: it converts an invisible dependency into a named failure.

5. Give every test a timeout

Without one, a hung test consumes the whole job budget and reports nothing:

pip install pytest-timeout

# pytest.ini
[pytest]
timeout = 60
timeout_method = thread
markers =
    slow: takes more than a second, or reads real data

@pytest.mark.timeout(300)          # this one is allowed longer
@pytest.mark.slow
def test_full_pipeline(real_sample):
    ...

Set a job timeout as well, so a runner cannot sit for six hours:

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 20

A per-test timeout names the offender; a job timeout bounds the cost. Both are worth having.

6. Keep fixtures from accumulating

A suite that passes test-by-test and fails as a whole is usually holding memory across tests:

# ❌ built once, mutated by whichever test runs first
@pytest.fixture(scope="session")
def big_gdf():
    return gpd.read_file("large.gpkg")

# ✅ fresh per test, and released afterwards
@pytest.fixture
def big_gdf():
    gdf = gpd.read_file("large.gpkg")
    yield gdf
    del gdf
    import gc; gc.collect()

Matplotlib is the other common accumulator, since every figure stays in a global registry until closed:

@pytest.fixture(autouse=True)
def close_figures():
    yield
    import matplotlib.pyplot as plt
    plt.close("all")

autouse=True applies it to every test without any of them asking, which is right for a cleanup that should never be forgotten. The same leak in a batch job is covered in how to batch-generate map images.

Find the offender by measuring:

pip install pytest-memray
pytest --memray --most-allocations=10

pytest-memray reports total and peak allocation and ranks the tests that allocated most. Example 2 builds a lighter version of the same thing out of a conftest.py, and runs it.

A workflow for a small runner and per-test memory

The data used on this page

The rasters in the examples are generated, on purpose: a test's fixture should be exactly as big as the test needs, and a random 16,000 × 16,000 grid is the honest stand-in for "a large DEM tile" when the question is memory. The OOM and CPU measurements were made on this site's build machine (24 threads, 64 GB) with systemd-run and taskset, as shown. The setup makes the tests/ folder the examples write into.

import sys
from pathlib import Path

Path("tests").mkdir(exist_ok=True)
Path("tests/__init__.py").touch()

Example 1: a workflow tuned for a small runner

name: tests
on: [push, pull_request]

env:
  # native library limits — the four lines that fix most CI failures
  GDAL_CACHEMAX: 256
  GDAL_DISABLE_READDIR_ON_OPEN: EMPTY_DIR
  PROJ_NETWORK: "OFF"
  OMP_NUM_THREADS: 1
  OPENBLAS_NUM_THREADS: 1
  MKL_NUM_THREADS: 1
  NUMEXPR_NUM_THREADS: 1
  PYTHONUNBUFFERED: 1

jobs:
  fast:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: prefix-dev/setup-pixi@v0.8.1
        with:
          locked: true
          cache: true

      - name: Report the environment
        run: |
          pixi run python -c "
          import geopandas, pyproj, rasterio, os, multiprocessing as mp
          from shapely import geos_version_string
          print(f'geopandas {geopandas.__version__}  GEOS {geos_version_string}')
          print(f'PROJ {pyproj.proj_version_str}  GDAL {rasterio.__gdal_version__}')
          print(f'cpus reported {mp.cpu_count()}  OMP {os.environ.get(\"OMP_NUM_THREADS\")}')
          print(f'PROJ network {pyproj.network.is_network_enabled()}')"
          free -m
          nproc

      - name: Fast tests
        run: >
          pixi run pytest -m "not slow"
          --timeout=60 --timeout-method=thread
          -p no:cacheprovider -q --durations=10

  slow:
    runs-on: ubuntu-latest
    timeout-minutes: 45
    needs: fast
    steps:
      - uses: actions/checkout@v4
      - uses: prefix-dev/setup-pixi@v0.8.1
        with: { locked: true, cache: true }
      - name: Slow tests, one at a time
        run: >
          pixi run pytest -m "slow"
          --timeout=600 -p no:cacheprovider -q -x --durations=0

Splitting the jobs is the structural decision. Fast tests give a signal in two minutes, and slow ones run afterwards only if the fast ones passed — so a broken commit does not spend forty-five minutes proving it.

--durations=10 prints the ten slowest tests on every run, so a test that quietly grows from one second to thirty is visible before it becomes a timeout. -p no:cacheprovider stops pytest writing .pytest_cache in a container where it serves no purpose.

The environment report at the start pays for itself the first time CI and local disagree. nproc versus OMP_NUM_THREADS in the same output makes oversubscription obvious.

Example 2: measuring memory per test

%%writefile tests/conftest.py
import os
import gc
import pytest

MEMORY_LIMIT_MB = int(os.environ.get("TEST_MEMORY_LIMIT_MB", "0"))

def _rss_mb():
    try:
        import psutil
        return psutil.Process().memory_info().rss / 1024 ** 2
    except ImportError:
        import resource, sys
        rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        return rss / 1024 if sys.platform != "darwin" else rss / 1024 ** 2

@pytest.fixture(autouse=True)
def track_memory(request):
    """Sample RSS while the test runs: the OOM killer acts on the peak, not the end state."""
    import threading
    import time
    gc.collect()
    before = _rss_mb()
    peak = [before]
    done = threading.Event()

    def sample():
        while not done.is_set():
            peak[0] = max(peak[0], _rss_mb())
            time.sleep(0.01)

    sampler = threading.Thread(target=sample, daemon=True)
    sampler.start()
    yield
    done.set()
    sampler.join()
    if peak[0] - before > 100:
        print(f"\n  ⚠ {request.node.name} peaked {peak[0] - before:,.0f} MB above its start "
              f"({before:,.0f} → {peak[0]:,.0f} MB)")
    if MEMORY_LIMIT_MB and peak[0] > MEMORY_LIMIT_MB:
        pytest.fail(f"{request.node.name}: RSS peaked at {peak[0]:,.0f} MB, over the "
                    f"{MEMORY_LIMIT_MB:,} MB limit — this test would be OOM-killed in CI")

@pytest.fixture(autouse=True)
def close_matplotlib_figures():
    yield
    import sys
    if "matplotlib.pyplot" in sys.modules:
        sys.modules["matplotlib.pyplot"].close("all")

@pytest.fixture(autouse=True, scope="session")
def cap_native_libraries():
    """Cap GDAL's cache and PROJ's network from inside the suite, in case env vars were missed."""
    import rasterio
    import pyproj
    pyproj.network.set_network_enabled(False)
    with rasterio.Env(GDAL_CACHEMAX=256):
        yield

Example 3 writes the tests, and runs them under it with a 1 GB limit.

The value here is turning a silent exit 137 into a named test failure with a number. Note that it samples memory during the test from a background thread. An earlier version compared memory before and after each test, and on the run below it reported nothing at all: a test's arrays are freed when it returns, so its end state looks innocent. The OOM killer acts on the peak, so the peak is what has to be measured. The threshold is set from an environment variable so developers can run without it and CI can enforce it.

Checking sys.modules before importing pyplot avoids pulling matplotlib into every test run that does not use it — importing it costs about a second.

Example 3: making a heavy test fit

%%writefile tests/test_zonal.py
import numpy as np
import pytest
import geopandas as gpd
import rasterio
from rasterio.features import geometry_mask
from rasterio.mask import mask
from rasterio.transform import from_origin
from shapely.geometry import box


def write_raster(path, size):
    data = np.random.default_rng(42).integers(0, 1000, (size, size)).astype("int16")
    profile = dict(driver="GTiff", height=size, width=size, count=1, dtype="int16", crs="EPSG:27700",
                   nodata=-9999, transform=from_origin(0, size, 1, 1), tiled=True, blockxsize=256, blockysize=256)
    with rasterio.open(path, "w", **profile) as dst:
        dst.write(data, 1)
    return path


def zones_for(size, n=8):
    step = size // (n + 1)
    return gpd.GeoDataFrame({"zone": range(n)}, crs=27700,
                            geometry=[box(i * step, i * step, i * step + step, i * step + step) for i in range(n)])


def zonal_means(raster_path, zones):
    """Window reads: only the pixels under each zone are ever in memory."""
    means = []
    with rasterio.open(raster_path) as src:
        for geom in zones.to_crs(src.crs).geometry:
            arr, _ = mask(src, [geom], crop=True, filled=False)
            vals = arr[0].compressed()
            means.append(float(vals.mean()) if vals.size else None)
    return means


@pytest.fixture(scope="module")
def large_raster(tmp_path_factory):
    """16,000 × 16,000 int16 — 512 MB of pixels, as a real DEM tile might be."""
    return write_raster(tmp_path_factory.mktemp("big") / "large.tif", 16_000)


@pytest.fixture
def small_raster(tmp_path):
    """256 × 256 — the same code path at 1/4,000th of the pixels."""
    return write_raster(tmp_path / "small.tif", 256)


def test_zonal_stats_whole_raster(large_raster):
    """The naive version: read everything, then mask per zone."""
    zones = zones_for(16_000)
    with rasterio.open(large_raster) as src:
        data = src.read(1, masked=True)                     # the whole array in memory
        results = [data[~geometry_mask([g], data.shape, src.transform)].mean() for g in zones.geometry]
    assert len(results) == len(zones)


def test_zonal_stats_windowed(large_raster):
    means = zonal_means(large_raster, zones_for(16_000))
    assert len(means) == 8 and all(0 <= m <= 1000 for m in means)


def test_zonal_stats_on_small_raster(small_raster):
    """The behaviour test: fast, needs no marker, catches the same bugs."""
    means = zonal_means(small_raster, zones_for(256))
    assert len(means) == 8 and all(0 <= m <= 1000 for m in means)

Run under Example 2's conftest.py, with a 1 GB limit:

!TEST_MEMORY_LIMIT_MB=1000 {sys.executable} -m pytest -q -s --tb=line -p no:cacheprovider --color=no -W ignore::PendingDeprecationWarning tests

.
  ⚠ test_zonal_stats_whole_raster peaked 1,226 MB above its start (189 → 1,416 MB)
E..
==================================== ERRORS ====================================
______________ ERROR at teardown of test_zonal_stats_whole_raster ______________
E   Failed: test_zonal_stats_whole_raster: RSS peaked at 1,416 MB, over the 1,000 MB limit — this test would be OOM-killed in CI
=========================== short test summary info ============================
ERROR tests/test_zonal.py::test_zonal_stats_whole_raster - Failed: test_zonal...
3 passed, 1 error in 3.51s

The run shows the difference. The whole-raster test is the one the tracker names, and it takes most of the suite's time; the windowed test reads the same file with a small fraction of the memory. The third test is the point. It exercises the same code path — open, mask, compress, mean — on a raster small enough that memory is irrelevant. Almost every "this test needs 2 GB" is really "this fixture is larger than the test requires".

Keep one windowed test on real data, marked slow, to prove the windowing works at scale. The rest should be fast enough that nobody thinks about them, which is the argument in test fixtures for GIS code.

Assumptions your laptop meets and the runner does not

CI failures that do not reproduce locally almost always come from a resource assumption that your laptop satisfies and the runner does not.

The most consequential is memory, and the specific mechanism is worth understanding. When a container exceeds its limit, the kernel's OOM killer terminates the process with SIGKILL. There is no exception, no traceback, no partial pytest report — the process simply stops, and CI reports exit 137. Everything that would tell you which test did it is lost, which is why the fix starts with measuring rather than guessing.

GDAL's cache is the commonest culprit because of a specific mismatch. GDAL_CACHEMAX defaults to a percentage of system memory, and GDAL reads that from the host, not from the cgroup limit that actually applies. On this site's 64 GB build machine the default came to 3.05 GiB — about 3.3 GB — so a 2 GB container on such a host gives GDAL a budget it is not allowed to use, and it will use it. Setting an explicit value costs one line and removes the whole class of failure.

Thread oversubscription is the same mismatch in a different resource. os.cpu_count() and the thread-pool defaults in OpenMP and BLAS read the host's CPU count, and a CPU quota — which is how Docker, Kubernetes and most container runners limit CPU — does not change what they read. On the build machine, a two-core quota left NumPy's OpenBLAS starting 24 threads, and the same workload took 8.5 seconds instead of 0.9: the threads spent their time waiting for each other's slices. A CPU affinity limit, which removes cores from view, did not have the problem. Each thread also has a stack, so oversubscription costs memory too, and the two symptoms often appear together.

Test isolation matters more in CI because everything runs in one process, in order. Locally you run one test file while iterating; CI runs all of them, so anything that leaks accumulates. A session-scoped fixture holding a large frame, an unclosed matplotlib figure, a GDAL dataset left open — each is invisible in isolation and additive over a suite. Function-scoped fixtures and autouse cleanup fixtures are the cheap structural answer.

And the network is a resource too. A test that downloads data is slow, flaky and dependent on something outside the repository. PROJ's on-demand grid fetching is the version of this that nobody notices, because it is triggered from inside a coordinate transformation rather than from any line you wrote. PROJ_NETWORK=OFF plus a vendored proj-data makes it deterministic; blocking sockets in a fixture makes any remaining network dependency fail with a message that names it.

The deeper point is that these constraints are not obstacles to work around — they are the production environment. A scheduled job runs in a container with a memory limit, a CPU quota and possibly no internet. A test suite that only passes on a 64 GB workstation is not testing the conditions the code will actually meet. Making it fit a small runner is the same work as making it fit a production container, which is the argument developed in how to build a small, fast Python GIS Docker image.

SIGKILL against SIGTERM, cgroups and xdist

• Exit 137 is SIGKILL, exit 143 is SIGTERM. The first is the OOM killer; the second is a cancellation or timeout.
• GDAL_CACHEMAX reads host memory, not the cgroup limit. Always set it explicitly in CI.
• Thread limits must be set before import. In env: in the workflow, not in conftest.py.
• multiprocessing.cpu_count() reports host CPUs; len(os.sched_getaffinity(0)) respects the quota on Linux.
• pytest-xdist multiplies memory by worker count. On a small runner, -n 2 can cause the OOM you were avoiding.
• --durations=10 surfaces a test that is quietly growing before it becomes a timeout.
• pytest-memray attributes allocations per test, which is the fastest way to find the offender.
• macOS runners report RSS in bytes, Linux in kilobytes, via resource.getrusage. psutil avoids the difference.
• actions/cache on a pixi or conda environment cuts several minutes from every run.
• A job timeout does not name the offending test. Add a per-test timeout as well.

Internal links

• GIS tests pass locally but fail in CI — the environment-difference cases
• How to run a Python GIS pipeline in CI with GitHub Actions — the workflow this tunes
• How to build a small, fast Python GIS Docker image — the same limits, in production
• Test fixtures for GIS code — smaller fixtures as the real fix
• Fixing memory errors in GeoPandas when working with large files — the underlying memory problem
• How to process a very large GeoPackage in chunks — the windowing technique
• Reproducible GIS environments explained — pinning so CI matches local
• How to test a GIS pipeline with pytest — the suite being run

FAQ

What does exit code 137 mean?

128 + 9, so SIGKILL — almost always the kernel's OOM killer. There is no traceback because the process was terminated without warning. Cap GDAL_CACHEMAX and shrink the fixtures.

Why is CI twelve times slower than my laptop?

Usually thread oversubscription. Libraries size their thread pools from the host's CPU count, not the container's quota, so several of them each start 32 threads on 2 cores. Set OMP_NUM_THREADS=1 and the related variables.

What should GDAL_CACHEMAX be?

A few hundred megabytes for a test suite — 256 is a reasonable default. The point is that it is bounded and known, not that a specific number is optimal.

Why do tests pass individually but fail together?

Something is accumulating: a session-scoped fixture, unclosed matplotlib figures, open GDAL datasets. Use function scope and an autouse cleanup fixture, then measure with pytest-memray.

Should I use pytest-xdist to speed things up?

Carefully. Each worker is a separate process with its own memory, so -n 4 can quadruple peak usage and cause the OOM you were trying to avoid. On a small runner, fixing oversubscription usually helps more.

How do I find which test uses the memory?

pytest --memray --most-allocations=10, or an autouse fixture that measures RSS around each test and reports growth over a threshold.

Should tests download data?

No. Vendor small fixtures in the repository, and set PROJ_NETWORK=OFF so coordinate transformations cannot fetch grids mid-test. Blocking sockets in a fixture will find anything you missed.