GIS Tests Run Out of Memory or Time Out in CI: How to Fix It

Problem statement

The suite passes locally in ninety seconds. In GitHub Actions:

Error: The operation was canceled.
##[error]The job running on runner GitHub Actions 12 has exceeded the maximum
execution time of 360 minutes.

Or it dies without explanation:

##[error]Process completed with exit code 137.

Exit code 137 is 128 + 9 — the process received SIGKILL. Nothing in the logs says why, because the kernel's OOM killer does not ask permission and pytest never gets a chance to report.

Or it simply crawls: eighteen minutes for a suite that takes ninety seconds on a laptop, so nobody waits for it and the checks get merged around.

CI runners are small — GitHub's standard hosted runner has 2 cores and 7 GB of RAM — and GIS libraries are written on the assumption that memory is plentiful. Most of the gap between local and CI comes from three or four specific behaviours, all of which are fixable.

Quick answer

env:
  GDAL_CACHEMAX: 256          # MB — the default is a share of HOST memory
  OMP_NUM_THREADS: 1          # stop each library spawning a thread per core
  OPENBLAS_NUM_THREADS: 1
  NUMEXPR_NUM_THREADS: 1
  PROJ_NETWORK: OFF           # no grid downloads mid-test

pytest -m "not slow" -p no:cacheprovider --timeout=60 -x -q

Triage rows matching each CI symptom to its cause and fix. — Exit 137 is the kernel, not pytest. It never appears in the test output.

Symptom	Cause	Fix
exit code 137	OOM killed	cap `GDAL_CACHEMAX`, chunk the data, smaller fixtures
exit code 143	`SIGTERM` — job cancelled or timed out	a per-test timeout, and a job timeout
very slow, low CPU	downloading data or PROJ grids	vendor fixtures, `PROJ_NETWORK=OFF`
very slow, high CPU	thread oversubscription	pin the `*_NUM_THREADS` variables to 1
passes alone, fails in the suite	fixtures accumulating between tests	function scope, and free explicitly
flaky, no pattern	tests sharing state or a temp path	`tmp_path`, and `-p no:randomly` to check

Cap GDAL_CACHEMAX first. It is one line and it is the single most common cause of exit 137.

Step-by-step solution

1. Read the exit code

exit 137  =  128 + 9   → SIGKILL   → almost always the OOM killer
exit 143  =  128 + 15  → SIGTERM   → cancelled, or a timeout
exit 124                → the `timeout` command fired
exit 1                  → tests genuinely failed

Exit 137 means the kernel killed the process. pytest produced no report because it was not asked. Confirm it by watching memory as the job runs:

- name: Watch memory
  run: |
    ( while true; do
        free -m | awk '/^Mem:/ {printf "%s used=%sMB avail=%sMB\n", strftime("%H:%M:%S"), $3, $7}'
        sleep 5
      done ) &
    echo $! > /tmp/memwatch.pid

- name: Tests
  run: pytest -q

- name: Stop watching
  if: always()
  run: kill "$(cat /tmp/memwatch.pid)" 2>/dev/null || true

10:14:02 used=1204MB avail=5611MB
10:14:07 used=3891MB avail=2918MB
10:14:12 used=6702MB avail=118MB
##[error]Process completed with exit code 137.

The jump from 1.2 GB to 6.7 GB in ten seconds names the test that did it.

2. Cap `GDAL_CACHEMAX` — the commonest single cause

Panels contrasting GDAL sizing its cache from host memory against an explicit cap. — GDAL reads the host's memory, not the container's limit. In CI those are very different numbers.

GDAL's block cache defaults to a percentage of system memory — 5% in recent versions. On a 64 GB build host that is 3.2 GB, and GDAL will allocate it happily while the container is limited to 2 GB.

env:
  GDAL_CACHEMAX: 256        # MB, since GDAL 3.x accepts a plain number as MB

# or from Python, before opening anything
from osgeo import gdal
gdal.SetCacheMax(256 * 1024 * 1024)
print(f"GDAL cache: {gdal.GetCacheMax() / 1024**2:.0f} MB")

Two related settings matter for raster work:

  GDAL_SWATH_SIZE: 33554432          # 32 MB per warp swath, default is larger
  GDAL_DISABLE_READDIR_ON_OPEN: EMPTY_DIR   # do not list a whole directory per open
  VSI_CACHE: FALSE                   # no extra /vsi file cache in a short-lived job

GDAL_DISABLE_READDIR_ON_OPEN is a speed fix rather than a memory one, and a large one on network or object storage: without it, opening one file lists the entire directory.

3. Stop libraries oversubscribing the CPU

NumPy, GDAL, PROJ and BLAS each spawn worker threads sized from the CPU count they detect. A container limited to 2 cores on a 32-core host often reports 32, so four libraries each start 32 threads and 128 threads fight over 2 cores.

env:
  OMP_NUM_THREADS: 1
  OPENBLAS_NUM_THREADS: 1
  MKL_NUM_THREADS: 1
  NUMEXPR_NUM_THREADS: 1
  GDAL_NUM_THREADS: 1

# these must be set before the libraries are imported
import os
for var in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ.setdefault(var, "1")
import numpy, geopandas          # now they see the limit

before:  pytest -q   →  18m 42s,  load average 46.2
after:   pytest -q   →   2m 11s,  load average  1.9

Setting them in the workflow's env: block is more reliable than in conftest.py, because it applies before Python starts. Thread limits read at import time are ignored if set afterwards.

Each thread also has a stack and its own buffers, so oversubscription costs memory as well as time — which is why this sometimes fixes an exit 137 rather than only a slow job.

4. Do not download anything during tests

Network work in a test is slow, flaky, and occasionally silent:

# ❌ every run, over the network
@pytest.fixture
def uk_boundary():
    return gpd.read_file("https://example.org/uk.geojson")

# ✅ committed, tiny, offline
@pytest.fixture(scope="session")
def uk_boundary():
    return gpd.read_file(Path(__file__).parent / "data" / "uk_simplified.gpkg")

The subtler network dependency is PROJ. With PROJ_NETWORK=ON, a datum transformation may fetch a grid from a CDN mid-test — adding seconds, and making results depend on network state:

env:
  PROJ_NETWORK: OFF

def test_no_network_during_tests(monkeypatch):
    """Fail loudly if anything tries to open a socket."""
    import socket
    def blocked(*args, **kwargs):
        raise RuntimeError("a test attempted a network connection")
    monkeypatch.setattr(socket.socket, "connect", blocked)
    run_the_pipeline_under_test()

Blocking sockets is a blunt instrument and an effective one: it converts an invisible dependency into a named failure.

5. Give every test a timeout

Without one, a hung test consumes the whole job budget and reports nothing:

pip install pytest-timeout

# pytest.ini
[pytest]
timeout = 60
timeout_method = thread
markers =
    slow: takes more than a second, or reads real data

@pytest.mark.timeout(300)          # this one is allowed longer
@pytest.mark.slow
def test_full_pipeline(real_sample):
    ...

Set a job timeout as well, so a runner cannot sit for six hours:

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 20

A per-test timeout names the offender; a job timeout bounds the cost. Both are worth having.

6. Keep fixtures from accumulating

A suite that passes test-by-test and fails as a whole is usually holding memory across tests:

# ❌ built once, mutated by whichever test runs first
@pytest.fixture(scope="session")
def big_gdf():
    return gpd.read_file("large.gpkg")

# ✅ fresh per test, and released afterwards
@pytest.fixture
def big_gdf():
    gdf = gpd.read_file("large.gpkg")
    yield gdf
    del gdf
    import gc; gc.collect()

Matplotlib is the other common accumulator, since every figure stays in a global registry until closed:

@pytest.fixture(autouse=True)
def close_figures():
    yield
    import matplotlib.pyplot as plt
    plt.close("all")

autouse=True applies it to every test without any of them asking, which is right for a cleanup that should never be forgotten. The same leak in a batch job is covered in how to batch-generate map images.

Find the offender by measuring:

pip install pytest-memray
pytest --memray --most-allocations=10

📦 Total memory allocated: 4.2GiB
📈 Peak memory usage: 3.9GiB
  tests/test_raster.py::test_zonal_stats           2.1GiB
  tests/test_clean.py::test_dissolve_national      1.4GiB

Code examples

Example 1: a workflow tuned for a small runner

name: tests
on: [push, pull_request]

env:
  # native library limits — the four lines that fix most CI failures
  GDAL_CACHEMAX: 256
  GDAL_DISABLE_READDIR_ON_OPEN: EMPTY_DIR
  PROJ_NETWORK: "OFF"
  OMP_NUM_THREADS: 1
  OPENBLAS_NUM_THREADS: 1
  MKL_NUM_THREADS: 1
  NUMEXPR_NUM_THREADS: 1
  PYTHONUNBUFFERED: 1

jobs:
  fast:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: prefix-dev/setup-[email protected]
        with:
          locked: true
          cache: true

      - name: Report the environment
        run: |
          pixi run python -c "
          import geopandas, pyproj, rasterio, os, multiprocessing as mp
          from shapely import geos_version_string
          print(f'geopandas {geopandas.__version__}  GEOS {geos_version_string}')
          print(f'PROJ {pyproj.proj_version_str}  GDAL {rasterio.__gdal_version__}')
          print(f'cpus reported {mp.cpu_count()}  OMP {os.environ.get(\"OMP_NUM_THREADS\")}')
          print(f'PROJ network {pyproj.network.is_network_enabled()}')"
          free -m
          nproc

      - name: Fast tests
        run: >
          pixi run pytest -m "not slow"
          --timeout=60 --timeout-method=thread
          -p no:cacheprovider -q --durations=10

  slow:
    runs-on: ubuntu-latest
    timeout-minutes: 45
    needs: fast
    steps:
      - uses: actions/checkout@v4
      - uses: prefix-dev/setup-[email protected]
        with: { locked: true, cache: true }
      - name: Slow tests, one at a time
        run: >
          pixi run pytest -m "slow"
          --timeout=600 -p no:cacheprovider -q -x --durations=0

Splitting the jobs is the structural decision. Fast tests give a signal in two minutes, and slow ones run afterwards only if the fast ones passed — so a broken commit does not spend forty-five minutes proving it.

--durations=10 prints the ten slowest tests on every run, so a test that quietly grows from one second to thirty is visible before it becomes a timeout. -p no:cacheprovider stops pytest writing .pytest_cache in a container where it serves no purpose.

The environment report at the start pays for itself the first time CI and local disagree. nproc versus OMP_NUM_THREADS in the same output makes oversubscription obvious.

Example 2: measuring memory per test

# tests/conftest.py
import os
import gc
import pytest

MEMORY_LIMIT_MB = int(os.environ.get("TEST_MEMORY_LIMIT_MB", "0"))

def _rss_mb():
    try:
        import psutil
        return psutil.Process().memory_info().rss / 1024 ** 2
    except ImportError:
        import resource, sys
        rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        return rss / 1024 if sys.platform != "darwin" else rss / 1024 ** 2

@pytest.fixture(autouse=True)
def track_memory(request):
    gc.collect()
    before = _rss_mb()
    yield
    gc.collect()
    after = _rss_mb()
    grew = after - before
    if grew > 100:
        print(f"\n  ⚠ {request.node.name} grew RSS by {grew:.0f} MB "
              f"({before:.0f} → {after:.0f} MB)")
    if MEMORY_LIMIT_MB and after > MEMORY_LIMIT_MB:
        pytest.fail(f"{request.node.name}: RSS {after:.0f} MB exceeds the "
                    f"{MEMORY_LIMIT_MB} MB limit — this test would be OOM-killed in CI")

@pytest.fixture(autouse=True)
def close_matplotlib_figures():
    yield
    import sys
    if "matplotlib.pyplot" in sys.modules:
        sys.modules["matplotlib.pyplot"].close("all")

@pytest.fixture(autouse=True, scope="session")
def cap_native_libraries():
    """Cap GDAL's cache from inside the suite, in case env vars were missed."""
    try:
        from osgeo import gdal
        gdal.SetCacheMax(256 * 1024 ** 2)
    except ImportError:
        pass
    try:
        import pyproj
        pyproj.network.set_network_enabled(False)
    except ImportError:
        pass
    yield

TEST_MEMORY_LIMIT_MB=1500 pytest -q

  ⚠ test_dissolve_national grew RSS by 1,402 MB (184 → 1,586 MB)
FAILED tests/test_clean.py::test_dissolve_national - RSS 1586 MB exceeds the 1500 MB
limit — this test would be OOM-killed in CI

The value here is turning a silent exit 137 into a named test failure with a number. The threshold is set from an environment variable so developers can run without it and CI can enforce it.

Checking sys.modules before importing pyplot avoids pulling matplotlib into every test run that does not use it — importing it costs about a second.

Example 3: making a heavy test fit

import pytest
import geopandas as gpd
import numpy as np
import rasterio
from rasterio.windows import Window

# ── before: reads the whole raster, ~2.1 GB peak ────────────────────────────
@pytest.mark.slow
def test_zonal_stats_naive(large_raster, zones):
    with rasterio.open(large_raster) as src:
        data = src.read(1, masked=True)          # the whole array in memory
    results = [data[rows, cols].mean() for rows, cols in zone_indices(zones, src)]
    assert len(results) == len(zones)

# ── after: window reads, ~90 MB peak ────────────────────────────────────────
def test_zonal_stats_windowed(large_raster, zones):
    from rasterio.mask import mask
    means = []
    with rasterio.open(large_raster) as src:
        zones = zones.to_crs(src.crs)
        for geom in zones.geometry:
            arr, _ = mask(src, [geom], crop=True, filled=False)
            vals = arr[0].compressed()
            means.append(float(vals.mean()) if vals.size else None)
    assert len(means) == len(zones)
    assert all(m is None or 0 <= m <= 2000 for m in means)

# ── and a small fixture, so it is not "slow" at all ─────────────────────────
@pytest.fixture
def small_raster(tmp_path):
    """256×256 instead of 20000×20000 — same code path, 1/6000th the memory."""
    path = tmp_path / "small.tif"
    data = np.random.default_rng(42).integers(0, 1000, (256, 256)).astype("int16")
    profile = dict(driver="GTiff", height=256, width=256, count=1,
                   dtype="int16", crs="EPSG:27700", nodata=-9999,
                   transform=rasterio.transform.from_origin(0, 256, 1, 1))
    with rasterio.open(path, "w", **profile) as dst:
        dst.write(data, 1)
    return path

def test_zonal_stats_on_small_raster(small_raster, small_zones):
    """The behaviour test. Runs in 40 ms, needs no marker, catches the same bugs."""
    result = zonal_means(small_raster, small_zones)
    assert len(result) == len(small_zones)
    assert not any(np.isnan(v) for v in result if v is not None)

The third test is the point. It exercises the same code path — open, mask, compress, mean — on a raster small enough that memory is irrelevant. Almost every "this test needs 2 GB" is really "this fixture is larger than the test requires".

Keep one windowed test on real data, marked slow, to prove the windowing works at scale. The rest should be fast enough that nobody thinks about them, which is the argument in test fixtures for GIS code.

Explanation

Grid comparing what libraries detect from the host against the limits the container actually enforces. — Every one of these libraries reads the host and is bounded by the container.

CI failures that do not reproduce locally almost always come from a resource assumption that your laptop satisfies and the runner does not.

The most consequential is memory, and the specific mechanism is worth understanding. When a container exceeds its limit, the kernel's OOM killer terminates the process with SIGKILL. There is no exception, no traceback, no partial pytest report — the process simply stops, and CI reports exit 137. Everything that would tell you which test did it is lost, which is why the fix starts with measuring rather than guessing.

GDAL's cache is the commonest culprit because of a specific mismatch. GDAL_CACHEMAX defaults to a percentage of system memory, and GDAL reads that from the host, not from the cgroup limit that actually applies. A 64 GB build host with a 2 GB container gives GDAL a 3.2 GB budget it is not allowed to use, and it will use it. Setting an explicit value costs one line and removes the whole class of failure.

Thread oversubscription is the same mismatch in a different resource. multiprocessing.cpu_count() and the thread-pool defaults in OpenMP and BLAS read the host's CPU count, so a 2-core container on a 32-core host produces libraries each starting 32 workers. The result is 128 threads contending for 2 cores, with context-switching costing more than the work — which is why a suite can take twelve times longer in CI while doing exactly the same computation. It also costs memory, since every thread has a stack, so the two symptoms often appear together.

Test isolation matters more in CI because everything runs in one process, in order. Locally you run one test file while iterating; CI runs all of them, so anything that leaks accumulates. A session-scoped fixture holding a large frame, an unclosed matplotlib figure, a GDAL dataset left open — each is invisible in isolation and additive over a suite. Function-scoped fixtures and autouse cleanup fixtures are the cheap structural answer.

And the network is a resource too. A test that downloads data is slow, flaky and dependent on something outside the repository. PROJ's on-demand grid fetching is the version of this that nobody notices, because it is triggered from inside a coordinate transformation rather than from any line you wrote. PROJ_NETWORK=OFF plus a vendored proj-data makes it deterministic; blocking sockets in a fixture makes any remaining network dependency fail with a message that names it.

The deeper point is that these constraints are not obstacles to work around — they are the production environment. A scheduled job runs in a container with a memory limit, a CPU quota and possibly no internet. A test suite that only passes on a 64 GB workstation is not testing the conditions the code will actually meet. Making it fit a small runner is the same work as making it fit a production container, which is the argument developed in how to build a small, fast Python GIS Docker image.

Edge cases or notes

Exit 137 is SIGKILL, exit 143 is SIGTERM. The first is the OOM killer; the second is a cancellation or timeout.
GDAL_CACHEMAX reads host memory, not the cgroup limit. Always set it explicitly in CI.
Thread limits must be set before import. In env: in the workflow, not in conftest.py.
multiprocessing.cpu_count() reports host CPUs; len(os.sched_getaffinity(0)) respects the quota on Linux.
pytest-xdist multiplies memory by worker count. On a small runner, -n 2 can cause the OOM you were avoiding.
--durations=10 surfaces a test that is quietly growing before it becomes a timeout.
pytest-memray attributes allocations per test, which is the fastest way to find the offender.
macOS runners report RSS in bytes, Linux in kilobytes, via resource.getrusage. psutil avoids the difference.
actions/cache on a pixi or conda environment cuts several minutes from every run.
A job timeout does not name the offending test. Add a per-test timeout as well.

Internal links

GIS tests pass locally but fail in CI — the environment-difference cases
How to run a Python GIS pipeline in CI with GitHub Actions — the workflow this tunes
How to build a small, fast Python GIS Docker image — the same limits, in production
Test fixtures for GIS code — smaller fixtures as the real fix
Fixing memory errors in GeoPandas when working with large files — the underlying memory problem
How to process a very large GeoPackage in chunks — the windowing technique
Reproducible GIS environments explained — pinning so CI matches local
How to test a GIS pipeline with pytest — the suite being run

FAQ

What does exit code 137 mean?

128 + 9, so SIGKILL — almost always the kernel's OOM killer. There is no traceback because the process was terminated without warning. Cap GDAL_CACHEMAX and shrink the fixtures.

Why is CI twelve times slower than my laptop?

Usually thread oversubscription. Libraries size their thread pools from the host's CPU count, not the container's quota, so several of them each start 32 threads on 2 cores. Set OMP_NUM_THREADS=1 and the related variables.

What should `GDAL_CACHEMAX` be?

A few hundred megabytes for a test suite — 256 is a reasonable default. The point is that it is bounded and known, not that a specific number is optimal.

Why do tests pass individually but fail together?

Something is accumulating: a session-scoped fixture, unclosed matplotlib figures, open GDAL datasets. Use function scope and an autouse cleanup fixture, then measure with pytest-memray.

Should I use `pytest-xdist` to speed things up?

Carefully. Each worker is a separate process with its own memory, so -n 4 can quadruple peak usage and cause the OOM you were trying to avoid. On a small runner, fixing oversubscription usually helps more.

How do I find which test uses the memory?

pytest --memray --most-allocations=10, or an autouse fixture that measures RSS around each test and reports growth over a threshold.

Should tests download data?

No. Vendor small fixtures in the repository, and set PROJ_NETWORK=OFF so coordinate transformations cannot fetch grids mid-test. Blocking sockets in a fixture will find anything you missed.

GIS Tests Run Out of Memory or Time Out in CI: How to Fix It #

Problem statement #

Quick answer #

Step-by-step solution #

1. Read the exit code #

2. Cap GDAL_CACHEMAX — the commonest single cause #

3. Stop libraries oversubscribing the CPU #

4. Do not download anything during tests #

5. Give every test a timeout #

6. Keep fixtures from accumulating #

Code examples #

Example 1: a workflow tuned for a small runner #

Example 2: measuring memory per test #

Example 3: making a heavy test fit #

Explanation #

Edge cases or notes #

Internal links #

FAQ #

What does exit code 137 mean? #

Why is CI twelve times slower than my laptop? #

What should GDAL_CACHEMAX be? #

Why do tests pass individually but fail together? #

Should I use pytest-xdist to speed things up? #

How do I find which test uses the memory? #

Should tests download data? #