⚙️ How-To Testing, CI & Docker Files & formats Updated

How to Test a GIS Script That Reads and Writes Files

Problem statement

Most GIS code is file-handling code, and most of it is written to be untestable:

def clean_parcels():
    gdf = gpd.read_file("/data/raw/parcels_2026.gpkg")
    gdf = gdf[gdf.geometry.is_valid]
    gdf.to_file("/data/clean/parcels_clean.gpkg", driver="GPKG")

There is no way to test this without the real path existing. Run it in CI and it fails; run it on a colleague's laptop and it fails; run it twice and the second run silently overwrites the first result.

The naive test is worse than none:

def test_clean_parcels():
    clean_parcels()
    result = gpd.read_file("/data/clean/parcels_clean.gpkg")
    assert len(result) > 0

It depends on a machine, mutates shared state, cannot run in parallel with itself, and leaves the output behind for the next test to find.

The fix has two halves: structure the code so the I/O is at the edges, and use tmp_path for the I/O that remains.

Quick answer 

import geopandas as gpd
import pytest
from shapely.geometry import box

# separate the pure transformation from the file handling
def clean(gdf):                                    # ← testable without any file
    return gdf[gdf.geometry.is_valid & gdf.geometry.notna()].copy()

def clean_file(src, dst, driver="GPKG"):           # ← a thin, testable shell
    gdf = gpd.read_file(src)
    out = clean(gdf)
    out.to_file(dst, driver=driver)
    return len(gdf), len(out)

def test_clean_is_pure():
    gdf = gpd.GeoDataFrame({"id": [1, 2]},
                           geometry=[box(0, 0, 1, 1), None], crs=27700)
    assert len(clean(gdf)) == 1

def test_clean_file_round_trip(tmp_path):
    src = tmp_path / "in.gpkg"
    dst = tmp_path / "out.gpkg"
    gpd.GeoDataFrame({"id": [1, 2]},
                     geometry=[box(0, 0, 1, 1), box(2, 2, 3, 3)],
                     crs=27700).to_file(src, driver="GPKG")

    n_in, n_out = clean_file(src, dst)

    assert dst.exists()
    back = gpd.read_file(dst)
    assert len(back) == n_out == 2
    assert back.crs.to_epsg() == 27700
Flow showing a thin read shell, a pure transformation core, and a thin write shell.
Push the file handling to the edges and the interesting code becomes testable without files at all.
Pattern Effect
pure function on a GeoDataFrame most tests need no file at all
paths as arguments, never constants the test decides where things go
tmp_path fixture a fresh directory per test, cleaned up
write to a temp name, then rename a crash never leaves a half-file

Step-by-step solution

1. Take paths as arguments

A hard-coded path makes a function untestable and unreusable in one stroke:

# ❌
def clean_parcels():
    gdf = gpd.read_file("/data/raw/parcels.gpkg")

# ✅
def clean_parcels(src, dst, *, driver="GPKG"):
    gdf = gpd.read_file(src)
    ...

Accept str or Path, and normalise once:

from pathlib import Path

def clean_parcels(src, dst, *, driver="GPKG"):
    src, dst = Path(src), Path(dst)
    if not src.exists():
        raise FileNotFoundError(f"input not found: {src}")
    dst.parent.mkdir(parents=True, exist_ok=True)
    ...

dst.parent.mkdir(parents=True, exist_ok=True) is worth having: without it, the function fails when the output directory does not exist yet, which in a test is a distracting failure and in production is a 3 a.m. one.

2. Use tmp_path, not a fixed temp directory 

def test_writes_geopackage(tmp_path):
    dst = tmp_path / "out.gpkg"
    write_something(dst)
    assert dst.exists()
    assert dst.stat().st_size > 0

tmp_path is a pathlib.Path to a fresh, per-test directory. pytest creates it, keeps the last few runs for inspection, and deletes older ones. Nothing to clean up, and two tests cannot collide.

Fixture Scope Use
tmp_path one test the default
tmp_path_factory session a shared read-only input built once
tmpdir one test the legacy py.path version — prefer tmp_path 
@pytest.fixture(scope="session")
def sample_gpkg(tmp_path_factory):
    """Built once for the whole run. Tests must not modify it."""
    path = tmp_path_factory.mktemp("data") / "sample.gpkg"
    make_sample_gdf().to_file(path, driver="GPKG")
    return path

Session scope is worth it when writing the file is slow. The discipline it demands is that no test mutates it — one that does corrupts every later test in a way that depends on execution order.

3. Test the round trip, not just that a file appeared

dst.exists() proves a file was created, not that it is right. Formats lose things:

def test_round_trip_preserves_what_matters(tmp_path):
    original = gpd.GeoDataFrame(
        {"id": [1, 2],
         "long_column_name_over_ten_chars": ["a", "b"],
         "when": pd.to_datetime(["2026-01-01", "2026-06-15"])},
        geometry=[box(0, 0, 1, 1), box(2, 2, 3, 3)],
        crs=27700)

    path = tmp_path / "out.gpkg"
    original.to_file(path, driver="GPKG")
    back = gpd.read_file(path)

    assert len(back) == len(original)
    assert back.crs == original.crs
    assert set(back.columns) == set(original.columns)
    assert back["long_column_name_over_ten_chars"].tolist() == ["a", "b"]
    for a, b in zip(original.geometry, back.geometry):
        assert a.equals(b)

The same test against a shapefile fails on the column name, because the format truncates to ten characters. That is a genuine finding about the format, and it belongs in a test:

@pytest.mark.parametrize("driver,suffix", [("GPKG", ".gpkg"), ("GeoJSON", ".geojson")])
def test_round_trip_by_format(tmp_path, driver, suffix):
    ...

def test_shapefile_truncates_column_names(tmp_path):
    """Documented shapefile behaviour — asserted so a format change is noticed."""
    path = tmp_path / "out.shp"
    gdf = gpd.GeoDataFrame({"a_very_long_column": [1]},
                           geometry=[box(0, 0, 1, 1)], crs=27700)
    with pytest.warns(UserWarning):
        gdf.to_file(path)
    assert "a_very_lon" in gpd.read_file(path).columns

Note pytest.warns rather than suppressing the warning: the test asserts that the warning happens, which means removing it would fail the test. See why are my shapefile column names truncated.

4. Write atomically, and test that you do

A half-written output is worse than none, because it looks complete:

def write_atomic(gdf, dst, driver="GPKG", **kwargs):
    """Write to a temporary name in the same directory, then rename."""
    dst = Path(dst)
    dst.parent.mkdir(parents=True, exist_ok=True)
    tmp = dst.with_name(f".{dst.name}.tmp")
    try:
        gdf.to_file(tmp, driver=driver, **kwargs)
        tmp.replace(dst)                 # atomic on the same filesystem
    finally:
        if tmp.exists():
            tmp.unlink()
    return dst
def test_failed_write_leaves_no_partial_file(tmp_path, monkeypatch):
    dst = tmp_path / "out.gpkg"

    def exploding_to_file(self, *a, **kw):
        Path(a[0]).write_bytes(b"partial")     # simulate a partial write
        raise OSError("disk full")

    monkeypatch.setattr(gpd.GeoDataFrame, "to_file", exploding_to_file)

    with pytest.raises(OSError):
        write_atomic(make_sample_gdf(), dst)

    assert not dst.exists(), "a failed write must not leave an output file"
    assert list(tmp_path.iterdir()) == [], "and must not leave a temp file"

Path.replace is atomic within one filesystem, so a reader either sees the old file or the complete new one, never a partial. tmp deliberately lives in the same directory as dst — a rename across filesystems is a copy, and not atomic.

5. Test error paths, not only the happy one 

def test_missing_input_raises_clearly(tmp_path):
    with pytest.raises(FileNotFoundError, match="input not found"):
        clean_parcels(tmp_path / "nope.gpkg", tmp_path / "out.gpkg")

def test_corrupt_input_fails_with_the_path_in_the_message(tmp_path):
    bad = tmp_path / "bad.gpkg"
    bad.write_bytes(b"not a geopackage")
    with pytest.raises(Exception) as exc:
        clean_parcels(bad, tmp_path / "out.gpkg")
    assert "bad.gpkg" in str(exc.value)

def test_empty_input_produces_empty_output(tmp_path):
    src, dst = tmp_path / "empty.gpkg", tmp_path / "out.gpkg"
    gpd.GeoDataFrame({"id": []}, geometry=[], crs=27700).to_file(src, driver="GPKG")
    clean_parcels(src, dst)
    assert gpd.read_file(dst).empty

def test_output_directory_is_created(tmp_path):
    dst = tmp_path / "nested" / "deeper" / "out.gpkg"
    clean_parcels(make_input(tmp_path), dst)
    assert dst.exists()

Empty input is the case that breaks most pipelines — an empty GeoDataFrame has no geometry type, so to_file can fail or write a file with no layer. Deciding what should happen and asserting it is the whole point.

6. Avoid mocking the file system

unittest.mock on gpd.read_file is tempting and usually a mistake:

# ❌ this tests that you called a function, not that the code works
@mock.patch("geopandas.read_file")
def test_clean(mock_read):
    mock_read.return_value = make_sample_gdf()
    clean_parcels("anything.gpkg", "out.gpkg")
    mock_read.assert_called_once()

That passes if the driver is wrong, the CRS is lost, the geometry type is unsupported, or the file cannot actually be written. Real temporary files are fast — a small GeoPackage writes in a few milliseconds — and they test the thing you care about.

Mock only what is genuinely external and slow: a network call, a database, a cloud bucket:

def test_uses_cached_download(tmp_path, monkeypatch):
    calls = []
    def fake_download(url, dest):
        calls.append(url)
        make_sample_gdf().to_file(dest, driver="GPKG")
    monkeypatch.setattr("mypipeline.download", fake_download)

    fetch_and_clean("https://example.org/data.gpkg", tmp_path)
    fetch_and_clean("https://example.org/data.gpkg", tmp_path)
    assert len(calls) == 1, "the second call should use the cache"

monkeypatch is preferable to mock.patch in pytest because it undoes itself at the end of the test automatically.

Vertical steps showing a write to a temporary name in the same directory followed by an atomic rename.
The output appears only when it is complete. A crash leaves nothing to mistake for a result.

Code examples

Example 1: fixtures for file-based testing 

# tests/conftest.py
import pytest
import geopandas as gpd
import pandas as pd
from shapely.geometry import box, Point

CRS = 27700

@pytest.fixture
def sample_gdf():
    """Five features with one of each awkward attribute type."""
    return gpd.GeoDataFrame(
        {
            "id": [1, 2, 3, 4, 5],
            "name": ["a", "b", "c", "d", "e"],
            "value": [1.5, 2.5, None, 4.5, 5.5],
            "when": pd.to_datetime(["2026-01-01", "2026-02-01", "2026-03-01",
                                    "2026-04-01", "2026-05-01"]),
            "flag": [True, False, True, False, True],
        },
        geometry=[box(i, i, i + 1, i + 1) for i in range(5)],
        crs=CRS,
    )

@pytest.fixture
def input_file(tmp_path, sample_gdf):
    """A GeoPackage on disk containing sample_gdf."""
    path = tmp_path / "input.gpkg"
    sample_gdf.to_file(path, driver="GPKG")
    return path

@pytest.fixture
def input_folder(tmp_path, sample_gdf):
    """A folder of five GeoPackages plus one file that is not spatial."""
    folder = tmp_path / "inputs"
    folder.mkdir()
    for i in range(5):
        sample_gdf.iloc[i:i + 1].to_file(folder / f"tile_{i:02d}.gpkg", driver="GPKG")
    (folder / "readme.txt").write_text("not a spatial file")
    return folder

@pytest.fixture
def corrupt_file(tmp_path):
    path = tmp_path / "corrupt.gpkg"
    path.write_bytes(b"SQLite format 3\x00" + b"\x00" * 200)   # header, no content
    return path

@pytest.fixture
def output_dir(tmp_path):
    d = tmp_path / "output"
    d.mkdir()
    return d

The readme.txt in input_folder is deliberate. A batch job that globs a folder will meet non-spatial files, and the test should say what happens — skipped, reported, or a failure. Without it in the fixture, that path is never exercised and the behaviour is whatever the code happens to do.

corrupt_file uses a valid SQLite header with no GeoPackage content, which produces a more realistic failure than random bytes: GDAL opens it and then finds no layers, which is exactly what a truncated download looks like.

Example 2: testing a folder-processing job 

import pytest
import geopandas as gpd
from pathlib import Path

def process_folder(src_dir, dst_dir, *, pattern="*.gpkg", on_error="report"):
    """Process every matching file; returns a per-file report."""
    src_dir, dst_dir = Path(src_dir), Path(dst_dir)
    dst_dir.mkdir(parents=True, exist_ok=True)
    results = []
    for path in sorted(src_dir.glob(pattern)):
        try:
            gdf = gpd.read_file(path)
            out = clean(gdf)
            write_atomic(out, dst_dir / path.name)
            results.append({"file": path.name, "status": "ok",
                            "rows_in": len(gdf), "rows_out": len(out)})
        except Exception as exc:
            if on_error == "raise":
                raise
            results.append({"file": path.name, "status": "failed",
                            "error": f"{type(exc).__name__}: {exc}"[:120]})
    return results

def test_processes_every_matching_file(input_folder, output_dir):
    results = process_folder(input_folder, output_dir)
    assert len(results) == 5, "readme.txt must not be picked up by *.gpkg"
    assert all(r["status"] == "ok" for r in results)
    assert len(list(output_dir.glob("*.gpkg"))) == 5

def test_one_bad_file_does_not_stop_the_run(input_folder, output_dir, corrupt_file):
    import shutil
    shutil.copy(corrupt_file, input_folder / "bad.gpkg")

    results = process_folder(input_folder, output_dir)

    ok = [r for r in results if r["status"] == "ok"]
    failed = [r for r in results if r["status"] == "failed"]
    assert len(ok) == 5, "the five good files must still be processed"
    assert len(failed) == 1
    assert failed[0]["file"] == "bad.gpkg"
    assert "bad.gpkg" not in {p.name for p in output_dir.glob("*")}

def test_on_error_raise_stops_immediately(input_folder, output_dir, corrupt_file):
    import shutil
    shutil.copy(corrupt_file, input_folder / "aaa_bad.gpkg")   # sorts first
    with pytest.raises(Exception):
        process_folder(input_folder, output_dir, on_error="raise")
    assert len(list(output_dir.glob("*.gpkg"))) == 0

def test_empty_folder_is_not_an_error(tmp_path, output_dir):
    empty = tmp_path / "empty"; empty.mkdir()
    assert process_folder(empty, output_dir) == []

def test_rerunning_is_idempotent(input_folder, output_dir):
    first = process_folder(input_folder, output_dir)
    sizes = {p.name: p.stat().st_size for p in output_dir.glob("*.gpkg")}
    second = process_folder(input_folder, output_dir)
    assert [r["status"] for r in first] == [r["status"] for r in second]
    assert {p.name: p.stat().st_size for p in output_dir.glob("*.gpkg")} == sizes

Six tests covering the behaviours that matter for a batch job: it processes what it should, it skips what it should not, one bad file does not sink the run, the strict mode really is strict, an empty folder is not an error, and re-running changes nothing. That last one is idempotency, and it is the property that makes a scheduled job safe to retry.

aaa_bad.gpkg is named to sort first so the raise test genuinely stops before any successful output, rather than passing by accident.

Example 3: a golden-file test with a regeneration switch

Some outputs are easier to compare against a known-good file than to describe in assertions:

import pytest
import geopandas as gpd
from pathlib import Path
from tests.geometry_assertions import assert_frames_match

GOLDEN = Path(__file__).parent / "golden"

@pytest.fixture
def regenerate(request):
    return request.config.getoption("--regenerate-golden")

def pytest_addoption(parser):            # in conftest.py
    parser.addoption("--regenerate-golden", action="store_true",
                     help="overwrite golden files with current output")

def test_pipeline_output_matches_golden(input_file, tmp_path, regenerate):
    out = tmp_path / "result.gpkg"
    run_pipeline(input_file, out)
    result = gpd.read_file(out)

    golden = GOLDEN / "pipeline_result.gpkg"
    if regenerate:
        golden.parent.mkdir(parents=True, exist_ok=True)
        result.to_file(golden, driver="GPKG")
        pytest.skip(f"regenerated {golden}")

    if not golden.exists():
        pytest.fail(f"{golden} missing — run pytest --regenerate-golden")

    expected = gpd.read_file(golden)
    assert_frames_match(result, expected, key="id", tolerance=1e-9)
pytest tests/test_pipeline.py                      # compare
pytest tests/test_pipeline.py --regenerate-golden  # accept the current output

Golden files catch changes no hand-written assertion would think to check — a column reordered, a CRS subtly altered, a geometry type changed. The risk is that regenerating becomes reflexive, so the file records whatever the code does rather than what it should do.

Two things keep that honest. Regeneration requires an explicit flag, so it cannot happen by accident. And the golden file is committed, so the diff appears in code review — which is where a reviewer can ask whether the change was intended. That review step is the entire value; without it a golden test is a way of writing down bugs.

Keep golden files small. A 200-feature GeoPackage is a reviewable artefact; a 200 MB one is a binary blob nobody will look at.

Explanation

Grid showing what each file format loses in a write-and-read round trip.
None of these losses are visible in memory. All of them are visible after a round trip.

Testing file-handling code is really a design problem, and the tests are the thing that reveals it.

Code that reads a fixed path, transforms, and writes a fixed path has no seams. There is nothing to call with different inputs, no return value to inspect, and no way to run it without the paths existing. The commonest response — mocking the file system — tests the calls rather than the behaviour, and passes when the driver is wrong or the CRS is lost.

The better response is hexagonal: push I/O to the edges and keep a pure core. clean(gdf) -> gdf is a function you can call with any input, that returns something you can assert on, that has no side effects and no environment dependencies. Most of the interesting logic belongs there, and most tests then need no file at all. clean_file(src, dst) becomes a thin shell whose remaining behaviour — reading, writing, creating directories, handling failure — is genuinely about files and worth testing separately.

tmp_path is what makes the remaining file tests safe. Each test gets a fresh directory, so nothing collides, nothing leaks between tests, and tests can run in parallel. pytest keeps the last few runs' directories, so a failure can be inspected afterwards, and deletes older ones. The alternative — a fixed /tmp/test_output — produces tests that pass in isolation, fail in sequence, and fail differently depending on order.

Round-trip tests earn their place because file formats lose things. A shapefile truncates column names to ten characters, cannot store a NULL, and has no reliable encoding declaration. GeoJSON coerces everything to WGS 84 and has no schema. GeoPackage keeps almost everything but normalises geometry through WKB. None of this is visible in memory, and all of it is visible after a write and a read. Asserting the round trip is how you discover which of your data survives the format you chose — the comparison in GIS vector file formats compared.

Atomic writes deserve a test of their own, because the failure they prevent is silent. A process killed mid-write leaves a file that exists, has a plausible size, and opens — and downstream code will read it as if it were complete. Writing to a temporary name in the same directory and renaming makes the output appear only when it is finished, since rename within a filesystem is atomic. Testing it requires simulating a failure, which is what monkeypatch is genuinely good for: not replacing the file system, but making one specific operation fail on demand.

Finally, the error paths matter more here than in most code, because file I/O is where the environment intrudes. Missing inputs, corrupt downloads, unwritable directories, empty results and permission failures are all normal conditions in a scheduled job, and each one has a right answer that only a test can pin down. A pipeline whose happy path is tested and whose failure modes are not is a pipeline that works until the first bad day — which is the argument made at greater length in what to test in a GIS pipeline.

Edge cases or notes

  • tmp_path is a pathlib.Path; tmpdir is the legacy py.path. Prefer tmp_path.
  • tmp_path_factory gives session scope, for an expensive input built once — but nothing may mutate it.
  • Path.replace is atomic only within one filesystem. Keep the temp file in the destination directory.
  • Empty GeoDataFrames have no geometry type, so to_file may fail or produce a layerless file. Decide and assert.
  • Shapefiles write several files. dst.exists() on the .shp says nothing about the .prj; check that too.
  • pytest.warns asserts a warning happens — better than suppressing it, since removing the warning then fails the test.
  • monkeypatch undoes itself; mock.patch as a bare call does not.
  • Golden files must be committed and small, or the review step that justifies them does not happen.
  • --basetemp overrides where tmp_path lives, useful when the default temp directory is small or slow.
  • Parallel runs with pytest-xdist need per-test temp directories, which tmp_path already provides.

FAQ

How do I test a function with a hard-coded path?

You cannot — change it to take the path as an argument. That is the fix, and it makes the function reusable as a side effect.

Should I mock read_file and to_file?

No. Mocks test that you called a function, not that the file is correct. Real temporary files write in milliseconds and catch driver, CRS and schema problems that mocks cannot.

What is tmp_path?

A pytest fixture giving each test a fresh pathlib.Path directory. pytest creates it, retains the last few runs for inspection, and cleans up older ones. Two tests can never collide.

How do I test that a failed write leaves no output?

Write to a temporary name and rename on success, then use monkeypatch to make the write raise and assert that neither the output nor the temp file exists.

Do I need to test the round trip through a file?

Yes, if the format matters. Formats lose things — shapefile truncates column names, GeoJSON forces WGS 84 — and none of it is visible until you write and read back.

Are golden-file tests a good idea?

Yes if they are small, committed, and regenerated only behind an explicit flag, so the change appears in code review. Otherwise they become a record of whatever the code currently does.

How do I test a folder-processing job?

Build a fixture folder containing the good files, a non-matching file and a corrupt one, then assert what gets processed, what gets skipped, what gets reported, and that re-running changes nothing.