🔧 Fix Batch processing Testing, CI & Docker Notebook By · Updated

Batch Job Gives Different Results Each Run: How to Fix It

On this page

Problem statement

Nothing changed. The output did.

import hashlib
import time

cleaned = run_pipeline(parcels)                      # HM Land Registry's Leeds parcels — see the setup
for run in (1, 2):
    cleaned.to_file(f"summary_{run}.gpkg")
    print(f"run {run}: sha256 {hashlib.sha256(open(f'summary_{run}.gpkg', 'rb').read()).hexdigest()[:12]}")
    time.sleep(1.1)
run 1: sha256 84aaf3cd44d2
run 2: sha256 7bd620d45b56

Same frame, written twice a second apart, different bytes. Sometimes that is all it is — a timestamp in the file, harmless. Sometimes it is the data. Here is the same pipeline run on the same parcels, once in the order the Land Registry supplies them and once in the reverse order — as it would see them if next month's file were sorted differently:

as_supplied = run_pipeline(parcels)
reordered = run_pipeline(parcels.iloc[::-1])
for name, out in [("as supplied", as_supplied), ("reordered", reordered)]:
    print(f"{name:<12} {len(out):,} rows, total area {out.area.sum():,.1f} m²")
changed = set(as_supplied["INSPIREID"]) ^ set(reordered["INSPIREID"])
print(f"titles in one output but not the other: {len(changed):,}")
as supplied  292,939 rows, total area 585,700,065.4 m²
reordered    292,939 rows, total area 585,700,065.4 m²
titles in one output but not the other: 1,096

Same row count, same total area to the square metre — every check a pipeline usually makes passes — and yet 1,096 different titles are in the output — 548 swapped for their twins. Each of those is a title whose polygon is identical to another's, and the pipeline keeps whichever it meets first.

A job that produces different answers from the same input cannot be tested, cannot be verified against yesterday, and cannot be trusted when a number looks wrong — because "run it again" is no longer a way to check anything.

Non-determinism in GIS pipelines has a small number of causes, and almost all of them are an unordered collection being consumed as if it were ordered.

Quick answer

Find the source by bisecting on the output, then remove it:

import hashlib

import geopandas as gpd


def fingerprint(gdf, *, sort_by=None):
    """A hash of the DATA, insensitive to row order and file encoding."""
    g = gdf.copy()
    if sort_by:
        g = g.sort_values(sort_by)
    g = g.sort_index(axis=1)                                   # column order
    wkb = g.geometry.to_wkb()
    attrs = g.drop(columns=g.geometry.name).astype(str)
    payload = (attrs.agg("|".join, axis=1) + "|" + wkb.map(bytes.hex)).sort_values()
    return hashlib.blake2b("\n".join(payload).encode(), digest_size=16).hexdigest()

a = fingerprint(as_supplied, sort_by="INSPIREID")
b = fingerprint(reordered, sort_by="INSPIREID")
print("deterministic" if a == b else "NON-DETERMINISTIC")
NON-DETERMINISTIC
Cause Symptom Fix
unsorted glob / rglob files processed in different order sorted()
set or dict iteration groups or keys in different order sort before iterating
drop_duplicates without sort a different row survives each run sort first, then drop
parallel results collected as they finish row order varies sort the results, or key them
dissolve / groupby on unordered input aggregation of first differs sort before grouping
floating-point summation order totals differ in the last digits sort, or use a stable reducer
a timestamp written into the output bytes differ, data does not strip or fix it
Path.iterdir() filesystem order, varies by machine sorted() 
# the four lines that fix most of it
for path in sorted(src.rglob("*.gpkg")):            # not rglob alone
    ...
gdf = gdf.sort_values("parcel_id").reset_index(drop=True)
gdf = gdf.loc[~gdf["parcel_id"].duplicated(keep="first")]     # after sorting
results = sorted(pool_results, key=lambda r: r["item"])

Ordered versus unordered

Collection types that guarantee order against those that do not. ordered, or only stable on this machineorder guaranteedsame on another machinesorted()✓yes✓yeslist✓yes✓yesdict (3.7+)✓insertionif insertion isset✕no✕norglob✕no✕noas_completed✕no✕noStable locally, different in CI — that is the signature spatialworkflow.io
Anything in the right-hand column produces a different answer on a different machine.

Bytes or data, bisect, sort everything

Each non-determinism symptom paired with its cause and fix. bisect on a data fingerprintSymptomFixdifferent files each runsorted() on every glob and iterdira different duplicate survivessort before drop_duplicatesrow order varies in parallelsort results, or use pool.maparea total moves in the lastdigitsfloat addition is not associative— roundbytes differ, data identicala timestamp in the header —compare dataAlmost all of it is an unordered collection consumed as ordered spatialworkflow.io
Bisect on the fingerprint. The step where two runs diverge is the step with the bug.

1. Separate "different bytes" from "different data"

# byte comparison — fails on harmless header timestamps
sha256sum output/a.gpkg

# data comparison — only fails when the data is genuinely different
fingerprint(gpd.read_file("output/a.gpkg"), sort_by="parcel_id")

A GeoPackage stores a creation timestamp, SQLite reorders pages, and shapefile writes a .shp header with its own metadata. All of that changes the bytes without changing a single feature. Chasing byte-level reproducibility first wastes a day; the data fingerprint answers the question that matters.

If byte-identical output is genuinely required — for a checksum-based cache, or a regulated delivery — the practical route is to normalise on read rather than to fight the writers:

def canonical_bytes(path, sort_by):
    gdf = gpd.read_file(path).sort_values(sort_by).reset_index(drop=True)
    return gdf.to_json(sort_keys=True).encode()      # deterministic serialisation

2. Bisect: fingerprint after every step

def traced(steps, gdf, sort_by):
    prints = []
    for name, fn in steps:
        gdf = fn(gdf)
        prints.append((name, fingerprint(gdf, sort_by=sort_by), len(gdf)))
    return gdf, prints

STEPS = [
    ("drop null geometry", lambda g: g[g.geometry.notna()]),
    ("make_valid", lambda g: g.set_geometry(g.geometry.make_valid())),
    ("deduplicate geometry", lambda g: g[~g.geometry.to_wkb().duplicated(keep="first")]),
    ("reset index", lambda g: g.reset_index(drop=True)),
]
_, a = traced(STEPS, parcels, "INSPIREID")
_, b = traced(STEPS, parcels.iloc[::-1], "INSPIREID")

for (name, ha, na), (_, hb, nb) in zip(a, b):
    mark = "  ← diverges here" if ha != hb else ""
    print(f"{name:24s} {ha[:12]}  {na:>7,}{mark}")
drop null geometry       85e3193249fc  293,729
make_valid               85e3193249fc  293,729
deduplicate geometry     73b218030db6  292,939  ← diverges here
reset index              73b218030db6  292,939  ← diverges here

One line of output, one line of code to look at. This beats reading the whole pipeline hoping to spot the problem.

3. Sort every discovery, without exception

# these all return filesystem order, which varies by OS, filesystem and inode reuse
glob.glob("data/*.gpkg")
Path("data").rglob("*.gpkg")
Path("data").iterdir()
os.listdir("data")

# always
sorted(Path("data").rglob("*.gpkg"))

This is the single most common cause. Filesystem order is not random — it is stable on one machine and different on another, which is why the job is deterministic locally and not in CI. See python glob is not finding all my shapefiles for the related discovery trap.

It matters even when files are processed independently, because anything that appends to a shared list, writes to a shared log, or assigns sequential ids inherits the order.

Deduplicating identical Leeds title polygons with keep-first, on the file as supplied and reversed, against a sorted rule. the checks pass, the answer changesrunrowstotal areatitles differkeep first, as supplied292,939585,700,065.4 m²—keep first, reversed292,939585,700,065.4 m²1,096lowest id first, eitherorder292,939585,700,065.4 m²0548 groups of identical polygons; keep-first picks whichever the file lists firstRow count and area cannot see which duplicate survived spatialworkflow.io
Measured on the Leeds file. Only comparing which ids survived shows the difference.

4. Sort before any operation that picks a winner

def dedupe_by_rule(g):
    """Deterministic: among identical polygons keep the lowest title id."""
    g = g.sort_values("INSPIREID")
    return g[~g.geometry.to_wkb().duplicated(keep="first")]


x = dedupe_by_rule(parcels)
y = dedupe_by_rule(parcels.iloc[::-1])
print(f"{len(x):,} and {len(y):,} rows; same titles: {set(x['INSPIREID']) == set(y['INSPIREID'])}")
292,939 and 292,939 rows; same titles: True

The same applies to every "pick one" operation:

gdf.dissolve(by="ward", aggfunc="first")          # first of WHAT order?
gdf.groupby("ward").first()
gpd.sjoin(...).loc[~index.duplicated(keep="first")]

first is only meaningful after a sort. Without one it means "whatever pandas encountered first", which depends on how the frame was built.

gdf = gdf.sort_values(["ward", "parcel_id"])
dissolved = gdf.dissolve(by="ward", aggfunc="first")     # now reproducible

5. Order parallel results before using them

from concurrent.futures import ProcessPoolExecutor, as_completed

# as_completed yields in completion order — genuinely non-deterministic
results = []
for fut in as_completed(futures):
    results.append(fut.result())

# fix: sort by a stable key from the item, not by arrival
results.sort(key=lambda r: r["item"])

# or use map, which preserves input order
with ProcessPoolExecutor(8) as pool:
    results = list(pool.map(process, sorted(items)))

as_completed is the right tool for progress reporting and the wrong one for building an ordered result. It is also a good example of how latent this bug can be: this site ran the Leeds pipeline four times through a six-worker pool collecting chunks as_completed, and got the same titles every time — because 547 of the 548 groups of identical polygons sit on adjacent rows of the file, so they always land in the same chunk. The code was order-dependent all along; the data just never exercised it. Use it, then sort — or use map when order matters more than early feedback.

6. Deal with floating-point summation order

import numpy as np

values = parcels.geometry.area.to_numpy()          # 293,729 real areas
a = values.sum()
b = values[::-1].sum()
print(f"{float(a)!r}\n{float(b)!r}\nequal: {a == b}")
587825898.4365298
587825898.4365296
equal: False

Floating-point addition is not associative, so a total genuinely depends on the order of the terms. In a parallel reduction the order varies with worker timing, and the last two digits of an area total move between runs.

This is usually harmless and occasionally not — a != comparison against a stored total, or a hash of the numbers, will flip.

# make it stable: sort before summing
total = float(np.sort(values).sum())

# or accept it, and compare with a tolerance instead of equality
assert abs(total - expected) < 1e-6 * abs(expected)

Rounding to a sensible precision at the boundary is usually the cleanest answer — areas to the square metre, lengths to the millimetre. The extra digits were never real.

7. Strip timestamps and machine identity from outputs

metadata = {
    "generated_at": datetime.now().isoformat(),      # differs every run
    "host": socket.gethostname(),                    # differs every machine
    "run_id": uuid.uuid4().hex,                      # differs always
    "source_hash": content_hash(src),                # ← the useful one
    "recipe": RECIPE,
}

Run metadata is worth keeping — see how to record run metadata and data lineage. Keep it in a sidecar file rather than inside the data, so the data can be compared and the provenance is still recorded.

output/
├── summary.gpkg          ← deterministic; compare this
└── summary.run.json      ← timestamps, host, run id

A determinism test and guard on Leeds titles

The data used on this page

HM Land Registry's INSPIRE Index Polygons for Leeds: 293,729 titles, among them 548 groups of titles whose polygons are exactly identical — the ingredient every order-dependence bug needs. The setup saves them as parcels.gpkg and writes pipeline.py, the small cleaning pipeline the page runs: drop nulls, repair, and keep the first of any identical polygons. Land Registry data © Crown copyright and database right 2026, OGL v3.

import sys
from pathlib import Path

import geopandas as gpd
import pyogrio
import requests

reply = requests.Session().get("https://use-land-property-data.service.gov.uk/datasets/inspire/download/Leeds.zip",
                               timeout=600)                    # sets a cookie, then redirects to itself
reply.raise_for_status()
Path("leeds_inspire.zip").write_bytes(reply.content)
pyogrio.set_gdal_config_options({"GML_DOWNLOAD_SCHEMA": "NO"})
parcels = gpd.read_file("zip://leeds_inspire.zip!Land_Registry_Cadastral_Parcels.gml", columns=["INSPIREID"])
parcels.to_file("parcels.gpkg")

Path("pipeline.py").write_text('''import hashlib


def run_pipeline(parcels):
    g = parcels[parcels.geometry.notna()]
    g = g.set_geometry(g.geometry.make_valid())
    g = g[~g.geometry.to_wkb().duplicated(keep="first")]      # keeps whichever comes first
    return g.reset_index(drop=True)


def fingerprint(gdf, *, sort_by=None):
    g = gdf.sort_values(sort_by) if sort_by else gdf
    g = g.sort_index(axis=1)
    wkb = g.geometry.to_wkb()
    attrs = g.drop(columns=g.geometry.name).astype(str)
    payload = (attrs.agg("|".join, axis=1) + "|" + wkb.map(bytes.hex)).sort_values()
    return hashlib.blake2b("\\n".join(payload).encode(), digest_size=16).hexdigest()
''')
from pipeline import run_pipeline

print(f"{len(parcels):,} titles; {parcels.geometry.to_wkb().duplicated().sum():,} are an exact copy of another's polygon")
293,729 titles; 790 are an exact copy of another's polygon

Example 1: a determinism test

%%writefile test_determinism.py
import geopandas as gpd
import pytest

from pipeline import fingerprint, run_pipeline


@pytest.fixture(scope="module")
def sample_input():
    return gpd.read_file("parcels.gpkg", rows=50_000)


def test_pipeline_is_deterministic(sample_input):
    a = run_pipeline(sample_input)
    b = run_pipeline(sample_input)
    assert fingerprint(a, sort_by="INSPIREID") == fingerprint(b, sort_by="INSPIREID")


def test_pipeline_is_order_independent(sample_input):
    """Same rows, reversed order — the output must not change."""
    a = run_pipeline(sample_input)
    b = run_pipeline(sample_input.iloc[::-1])
    assert fingerprint(a, sort_by="INSPIREID") == fingerprint(b, sort_by="INSPIREID")
!{sys.executable} -m pytest -q --no-header -p no:cacheprovider --color=no --tb=no -rf test_determinism.py
.F                                                                       [100%]
=========================== short test summary info ============================
FAILED test_determinism.py::test_pipeline_is_order_independent - AssertionErr...
1 failed, 1 passed in 4.81s

The second test is the valuable one, and here it is the one that fails. Running twice on one machine passes, because nothing about the input changed between the runs — which is exactly why "it gave the same answer twice" proves nothing. Reversing the order deliberately is what exposes a pipeline that depends on it.

Example 2: a determinism guard for the pipeline itself

def assert_deterministic_inputs(paths):
    """Refuse to run on a work list that is not stably ordered."""
    assert paths == sorted(paths), (
        "the work list is not sorted — output will depend on filesystem order"
    )

def assert_sorted_before(gdf, cols, operation):
    ordered = gdf[cols].reset_index(drop=True)
    assert ordered.equals(ordered.sort_values(cols).reset_index(drop=True)), (
        f"{operation} needs {cols} sorted first, or its result is arbitrary"
    )
try:
    assert_sorted_before(parcels, ["INSPIREID"], "deduplicate")
except AssertionError as err:
    print(f"AssertionError: {err}")
AssertionError: deduplicate needs ['INSPIREID'] sorted first, or its result is arbitrary

Cheap assertions at the point of risk, rather than a test that runs the whole pipeline twice. The Land Registry file is not sorted by title id, so a deduplication that trusts its row order is making an arbitrary choice.

Example 3: the fingerprint, as a run-over-run check

def compare_runs(current: gpd.GeoDataFrame, previous_path: Path, sort_by: str):
    if not previous_path.exists():
        return {"status": "no baseline"}
    previous = gpd.read_file(previous_path)
    fc, fp = fingerprint(current, sort_by=sort_by), fingerprint(previous, sort_by=sort_by)
    if fc == fp:
        return {"status": "identical"}
    return {
        "status": "changed",
        "rows": (len(previous), len(current)),
        "new_ids": sorted(set(current[sort_by]) - set(previous[sort_by]))[:10],
        "gone_ids": sorted(set(previous[sort_by]) - set(current[sort_by]))[:10],
    }


as_supplied.to_file("last_night.gpkg")
print(compare_runs(reordered, Path("last_night.gpkg"), "INSPIREID"))
print(compare_runs(dedupe_by_rule(parcels.iloc[::-1]).reset_index(drop=True),
                   Path("last_night.gpkg"), "INSPIREID"))
{'status': 'changed', 'rows': (292939, 292939), 'new_ids': [25288903, 25303196, 25330757, 25344961, 25356180, 25364678, 25368480, 25390437, 25393613, 25396435], 'gone_ids': [25288902, 25303197, 25330756, 25344962, 25356179, 25365162, 25368481, 25390436, 25393612, 25396436]}
{'status': 'changed', 'rows': (292939, 292939), 'new_ids': [25303196, 25344961, 25364678, 25368480, 25396434, 25410722, 25425602, 25441072, 25452275, 25463962], 'gone_ids': [25303197, 25344962, 25365162, 25368481, 25396436, 25410723, 25425606, 25441075, 25452276, 25463964]}

The first comparison reports a "change" that is only the reordering: the same rows, with titles swapped for their duplicates. The second compares a fixed run against last night's unfixed one and still differs, because last night's output was itself arbitrary — the baseline has to be regenerated once the rule is fixed. After that, any difference between last night and tonight is a real data change, and the job can report exactly what moved. That is impossible while the output is noisy.

Unordered collections consumed as ordered

Capabilities a deterministic pipeline unlocks. what determinism buysa golden test that stays greentestableany diff between runs is realdiffableunchanged input = unchanged outputcacheablere-running reproduces the bugdebuggableyesterday’s numbers can be checkedauditableNone of these are available while the output moves on its own spatialworkflow.io
Every one of these is impossible while the output varies for no reason.

Almost all non-determinism in a GIS pipeline traces to the same root: a collection that guarantees no order being consumed as if it did. Filesystem listings, Python sets, dict iteration in older versions, as_completed, thread scheduling — all of them are documented as unordered, and all of them appear ordered in casual testing because they are stable on one machine. That stability is what makes the bug so slippery: it reproduces perfectly until it runs somewhere else.

The consequences propagate. An unordered file list changes which row drop_duplicates keeps, which changes which geometry survives, which changes an area total, which changes a percentage in a report. A difference that started as inode ordering ends up as a number somebody has to explain.

Floating-point summation is the one genuinely unavoidable case. Addition is not associative in IEEE 754, so a parallel reduction over the same values in a different grouping produces a slightly different total — correctly. The fix is not to eliminate it but to stop depending on exact equality: round at the boundary to a precision the data actually supports, and compare with tolerances.

The payoff for fixing all of this is larger than "the hashes match". A deterministic pipeline can be tested (a golden result stays golden), diffed (any change between runs is a real change), cached (the incremental job can trust that unchanged input means unchanged output), and debugged (re-running reproduces the problem). None of those are available while the output moves on its own, which is why this is worth an afternoon even when the variation looks harmless.

Dicts, PYTHONHASHSEED and sjoin order

  • Python dicts preserve insertion order since 3.7, so dict iteration is deterministic given deterministic insertion. Sets never are.
  • PYTHONHASHSEED affects set iteration order across processes; it is randomised by default. Never rely on set order, even within one machine.
  • geopandas.sjoin result order depends on the index implementation, which can change between versions. Sort after joining.
  • dissolve sorts by the group key by default — that part is deterministic; aggfunc="first" over unsorted rows is not.
  • Multiprocessing on fork versus spawn changes module import order, which can change anything relying on import side effects.
  • GeoPackage stores a last_change timestamp in gpkg_contents, so byte-identical output requires rewriting it.
  • to_file on shapefile writes the current date into the .dbf header.
  • Sorting a large frame is not free — sort once at the boundary rather than before every operation.

FAQ

The bytes differ but the data looks the same. Does it matter?

Usually not — GeoPackage writes a creation timestamp and SQLite reorders pages. Compare a fingerprint of the data rather than the file, unless byte-identical output is a requirement.

Why is it deterministic on my machine and not in CI?

Filesystem ordering is stable per machine and differs between them. An unsorted glob reproduces perfectly locally and changes in CI.

Do I have to sort everything?

Sort discovery, and sort before any operation that picks a winner — drop_duplicates, first, deduplicating a join. Elsewhere it is unnecessary overhead.

Why does my area total change in the last two digits?

Floating-point addition is not associative, so a parallel reduction sums in a different order. Round at the boundary, and compare with a tolerance rather than for equality.

Does as_completed cause this?

Yes, if you build an ordered result from it. Use it for progress, then sort the results by a stable key — or use pool.map, which preserves input order.

How do I prove the pipeline is deterministic?

Run it twice with discovery order reversed, and at two different worker counts. Running it twice unchanged often passes by luck.

Is a golden test worth it?

Once the pipeline is deterministic, yes — it becomes the cheapest regression test available. Before then it fails constantly and gets deleted.