How to Build a Resumable Batch GIS Job in Python

You kick off a job that reprojects and clips 4,000 GeoPackages, go home, and come back to a traceback on file 2,600 — a disk filled up, a geometry blew up, or the machine simply rebooted overnight. Rerunning from scratch means burning hours redoing 2,599 files that were already perfect. This guide shows you how to build a batch job that remembers what it finished, so a crash halfway through costs you one file, not the whole run. You will add a skip guard, a manifest of completed work, atomic writes so a half-written file is never mistaken for done, and a retry pass for the failures.

Problem statement

You are running a long batch over a folder of vector files and hitting some mix of these symptoms:

A crash wipes out hours of progress — the job dies on one bad file at 70%, and restarting reprocesses everything from the beginning.
You cannot tell what finished — after a failure you have no record of which files completed cleanly and which were mid-write when the process died.
Half-written outputs look "done" — the job crashed while writing roads.gpkg, leaving a truncated file on disk; a naive "skip if output exists" check treats that corrupt stub as complete.
Failures get lost in the scroll — a handful of files error out among thousands, and their names vanish above the terminal buffer, so you never go back and fix them.
Reruns are not safe to repeat — running the script twice double-appends rows, duplicates features, or overwrites good output with garbage.

The goal: one script you can stop and restart at any time, that computes the remaining work from a durable record, writes each output atomically, logs failures for a separate retry, and produces identical results whether it runs once or five times.

Quick answer

Keep a manifest — a CSV or JSON file recording every input that has been handled and its status. At startup, read the manifest, subtract the done files from the folder listing, and process only what remains. For each file, write to a temporary path first and Path.replace() it into place so the final output only ever appears complete. Append a done or failed row to the manifest as each file finishes. Rerun the script and it picks up exactly where it stopped; run a retry pass and it reprocesses only the failed rows.

A resumable batch loop that checks a manifest, skips completed files, writes atomically, and records each result. — Check the manifest, skip what is already done, process the rest, then write atomically and record the result.

import csv
import geopandas as gpd
from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/clean")
out_dir.mkdir(parents=True, exist_ok=True)
manifest = Path("data/clean/_manifest.csv")

# 1. Read the manifest and build the set of inputs already done.
done = set()
if manifest.exists():
    with manifest.open(newline="") as fh:
        for row in csv.DictReader(fh):
            if row["status"] == "done":
                done.add(row["file"])

# 2. Remaining work = every input not already marked done.
todo = [p for p in sorted(src_dir.glob("*.gpkg")) if p.name not in done]
print(f"{len(done)} done, {len(todo)} to process")

# 3. Process each remaining file, writing atomically and recording status.
with manifest.open("a", newline="") as fh:
    writer = csv.writer(fh)
    for path in todo:
        try:
            gdf = gpd.read_file(path).to_crs("EPSG:4326")
            tmp = out_dir / f"{path.stem}.gpkg.tmp"
            final = out_dir / f"{path.stem}.gpkg"
            gdf.to_file(tmp, driver="GPKG")
            tmp.replace(final)                 # atomic: appears only when complete
            writer.writerow([path.name, "done", ""])
        except Exception as exc:
            writer.writerow([path.name, "failed", str(exc)])
        fh.flush()                             # persist each result immediately

That is the whole pattern. The rest of this article explains each decision — why a manifest beats an output-exists check, why the temp-then-rename dance matters, and how to make reruns genuinely safe.

Step-by-step solution

Start with the simplest guard: skip if the output exists

The cheapest resumability you can add is a single check at the top of the loop: if the output file is already there, move on. It requires no extra state and turns a full rerun into a fast skip over completed files.

import geopandas as gpd
from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/clean")
out_dir.mkdir(parents=True, exist_ok=True)

for path in sorted(src_dir.glob("*.gpkg")):
    out_path = out_dir / f"{path.stem}.gpkg"
    if out_path.exists():
        continue                       # already produced, skip
    gdf = gpd.read_file(path).to_crs("EPSG:4326")
    gdf.to_file(out_path, driver="GPKG")

This is a real improvement and worth having, but it is fragile on its own: if the job crashed while writing out_path, the file exists but is truncated, and this check happily skips it. We fix that with atomic writes and a manifest below.

Record progress in a manifest

An output-exists check can only ever answer "is there a file here?" A manifest answers the questions that actually matter: which inputs did we handle, when, and did they succeed? Keep it as a CSV (easy to eyeball and open in a spreadsheet) or JSON (easy to load as a dict). The manifest is the source of truth for what is done — the output files are just the product.

import csv
from pathlib import Path

manifest = Path("data/clean/_manifest.csv")

# Create the header once, on first run.
if not manifest.exists():
    with manifest.open("w", newline="") as fh:
        csv.writer(fh).writerow(["file", "status", "detail"])

Recording status — not just presence — is what lets you distinguish done, failed, and never attempted, which an existence check cannot.

Read the manifest at startup to compute the work-list

On every run, load the manifest first and turn it into a lookup, then filter the folder listing down to only the inputs that are not already done. This is the step that makes the job resumable: the remaining work is computed, not assumed.

import csv
from pathlib import Path

src_dir = Path("data/raw")
manifest = Path("data/clean/_manifest.csv")

done = set()
if manifest.exists():
    with manifest.open(newline="") as fh:
        for row in csv.DictReader(fh):
            if row["status"] == "done":
                done.add(row["file"])

todo = [p for p in sorted(src_dir.glob("*.gpkg")) if p.name not in done]
print(f"Skipping {len(done)} done, processing {len(todo)} remaining")

Note that failed rows are deliberately not added to done, so a failed file stays on the work-list and gets retried on the next run. If you would rather retry failures only in a dedicated pass, exclude them here too and handle them separately (see Example 5).

Write each output atomically

The dangerous window in any batch job is the moment a file is being written. If the process dies mid-write, you are left with a partial file that looks like a finished one. Avoid this entirely by writing to a temporary path and then renaming it into place with Path.replace(). A rename on the same filesystem is atomic: the final path either does not exist yet or points at a fully written file — never anything in between.

import geopandas as gpd
from pathlib import Path

def write_atomic(gdf, final_path):
    final_path = Path(final_path)
    tmp = final_path.with_suffix(final_path.suffix + ".tmp")
    gdf.to_file(tmp, driver="GPKG")     # all the risky I/O happens on the tmp
    tmp.replace(final_path)             # atomic swap into the real name
    return final_path

Because the real output name only ever appears after a complete write, your skip-if-exists guard becomes trustworthy again: a file at the final path is, by construction, a finished file.

Append a status row as each file finishes

Update the manifest per file, not once at the end — a job that only writes its record on completion loses everything if it crashes. Append a row and flush immediately so the result is on disk before the next file starts. Record failures too, with the error message, so nothing is silently dropped.

import csv
import geopandas as gpd
from pathlib import Path

src_dir, out_dir = Path("data/raw"), Path("data/clean")
manifest = Path("data/clean/_manifest.csv")

with manifest.open("a", newline="") as fh:
    writer = csv.writer(fh)
    for path in todo:                            # todo computed as above
        try:
            gdf = gpd.read_file(path).to_crs("EPSG:4326")
            write_atomic(gdf, out_dir / f"{path.stem}.gpkg")
            writer.writerow([path.name, "done", ""])
            print(f"OK   {path.name}")
        except Exception as exc:
            writer.writerow([path.name, "failed", str(exc)])
            print(f"FAIL {path.name}: {exc}")
        fh.flush()                               # each row hits disk right away

Add a retry pass for the failures

Because failures are recorded with their own status, you can reprocess just them without touching the thousands of files that already succeeded. Read the manifest, select the failed rows, and run the same processing over that short list (see Example 5 for a complete version).

import csv
from pathlib import Path

manifest = Path("data/clean/_manifest.csv")

failed = []
with manifest.open(newline="") as fh:
    for row in csv.DictReader(fh):
        if row["status"] == "failed":
            failed.append(row["file"])

print(f"{len(failed)} files to retry:", failed)

Code examples

Example 1: The simplest resumable guard — skip if output exists

The minimum viable resumable batch. Good for quick jobs where a crash mid-write is unlikely and you do not need a failure log.

import geopandas as gpd
from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/clean")
out_dir.mkdir(parents=True, exist_ok=True)

for path in sorted(src_dir.glob("*.gpkg")):
    out_path = out_dir / f"{path.stem}.gpkg"
    if out_path.exists():
        print(f"skip {path.name}")
        continue
    gdf = gpd.read_file(path).to_crs("EPSG:4326")
    gdf.to_file(out_path, driver="GPKG")
    print(f"done {path.name}")

Example 2: A JSON manifest of completed files, loaded to filter the work-list

JSON is convenient when you want to store richer metadata per file (row count, timestamp) and load the whole record as a dict in one line.

import json
import geopandas as gpd
from datetime import datetime, timezone
from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/clean")
out_dir.mkdir(parents=True, exist_ok=True)
manifest = Path("data/clean/_manifest.json")

# Load prior progress: {filename: {status, rows, finished}}.
record = json.loads(manifest.read_text()) if manifest.exists() else {}

todo = [p for p in sorted(src_dir.glob("*.gpkg"))
        if record.get(p.name, {}).get("status") != "done"]

for path in todo:
    gdf = gpd.read_file(path).to_crs("EPSG:4326")
    gdf.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")
    record[path.name] = {
        "status": "done",
        "rows": len(gdf),
        "finished": datetime.now(timezone.utc).isoformat(),
    }
    manifest.write_text(json.dumps(record, indent=2))   # rewrite after each file

Rewriting the whole JSON each iteration is fine for hundreds to low thousands of files. For very large batches, prefer the append-only CSV in Example 3, which does not grow more expensive as the manifest gets longer.

Example 3: Atomic write plus an append-only CSV manifest

The production pattern: Path.replace() for the output, and a manifest you only ever append to, flushed per file. This is the version to reach for on a real overnight job.

import csv
import geopandas as gpd
from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/clean")
out_dir.mkdir(parents=True, exist_ok=True)
manifest = Path("data/clean/_manifest.csv")

def write_atomic(gdf, final_path):
    tmp = final_path.with_suffix(final_path.suffix + ".tmp")
    gdf.to_file(tmp, driver="GPKG")
    tmp.replace(final_path)

# Build the done-set from any existing manifest.
done = set()
if manifest.exists():
    with manifest.open(newline="") as fh:
        done = {r["file"] for r in csv.DictReader(fh) if r["status"] == "done"}
else:
    with manifest.open("w", newline="") as fh:
        csv.writer(fh).writerow(["file", "status", "detail"])

todo = [p for p in sorted(src_dir.glob("*.gpkg")) if p.name not in done]

with manifest.open("a", newline="") as fh:
    writer = csv.writer(fh)
    for path in todo:
        try:
            gdf = gpd.read_file(path).to_crs("EPSG:4326")
            write_atomic(gdf, out_dir / f"{path.stem}.gpkg")
            writer.writerow([path.name, "done", ""])
        except Exception as exc:
            writer.writerow([path.name, "failed", str(exc)])
        fh.flush()

Example 4: Idempotent processing you can rerun safely

An idempotent job produces the same result no matter how many times you run it. The keys are: derive output names deterministically from the input, clean up any stale .tmp from a previous crash, and never append to existing output — always replace it.

import geopandas as gpd
from pathlib import Path

def process_one(src_path, out_dir):
    out_dir = Path(out_dir)
    final = out_dir / f"{src_path.stem}.gpkg"      # deterministic from input
    tmp = final.with_suffix(".gpkg.tmp")

    if tmp.exists():
        tmp.unlink()                               # drop a leftover partial write

    gdf = gpd.read_file(src_path).to_crs("EPSG:4326")
    gdf.to_file(tmp, driver="GPKG")
    tmp.replace(final)                             # replace, never append
    return final

Running this twice on the same input yields byte-for-byte the same output and leaves no duplicate features or stray temp files — which is exactly what makes a mid-run restart safe.

Example 5: A retry-only pass over the failed entries

Once the main run is done, reprocess just the failed files. On success, append a fresh done row; the newest status for a file wins when you rebuild the done-set, so a later done supersedes an earlier failed.

import csv
import geopandas as gpd
from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/clean")
manifest = Path("data/clean/_manifest.csv")

def write_atomic(gdf, final_path):
    tmp = final_path.with_suffix(final_path.suffix + ".tmp")
    gdf.to_file(tmp, driver="GPKG")
    tmp.replace(final_path)

# Latest status per file wins, so re-tried successes override old failures.
status = {}
with manifest.open(newline="") as fh:
    for row in csv.DictReader(fh):
        status[row["file"]] = row["status"]

retry = [name for name, st in status.items() if st == "failed"]
print(f"Retrying {len(retry)} previously failed files")

with manifest.open("a", newline="") as fh:
    writer = csv.writer(fh)
    for name in retry:
        path = src_dir / name
        try:
            gdf = gpd.read_file(path).to_crs("EPSG:4326")
            write_atomic(gdf, out_dir / f"{path.stem}.gpkg")
            writer.writerow([name, "done", "retry"])
        except Exception as exc:
            writer.writerow([name, "failed", str(exc)])
        fh.flush()

Explanation

Why "output exists" alone is fragile

An existence check answers the wrong question. It tells you a file is present, not that it is correct. If the job crashed mid-write, the operating system has already created the output path but only flushed part of the data — so the file exists, is corrupt, and your if out_path.exists(): continue guard skips it forever. You end the run believing that file succeeded. The existence check is a useful fast path, but only becomes trustworthy once every write is atomic, because then presence really does imply completeness.

What atomic writes actually guarantee

gdf.to_file(...) is not atomic: it opens the output, streams data, and closes it, and a crash at any point in that sequence leaves a partial file at the final name. Writing to output.gpkg.tmp and then calling tmp.replace(final) moves all that risk onto the temp name. Path.replace() maps to the operating system's rename call, which is atomic on a single filesystem — at every instant the final path either does not exist or names a complete file. There is no observable half-state. The one caveat is that the temp file and the final file must live on the same filesystem, or the "rename" becomes a copy-then-delete and loses atomicity; keeping the .tmp in the same output directory guarantees this.

Why idempotence is the real goal

Resumability and idempotence are two sides of one property: a job you can safely run again. If reprocessing a file yields the same output and the same manifest state as processing it the first time, then it does not matter where a crash happened — rerunning is always correct. That is why the patterns here derive output names deterministically from inputs, replace rather than append output, flush the manifest per file, and treat the newest status row as authoritative. Get idempotence right and "resume" is simply "run it again"; the job sorts out for itself what still needs doing.

Edge cases or notes

Leftover `.tmp` files after a crash

A crash mid-write leaves an orphaned .tmp in the output folder. These are harmless — the final name was never created, so the file is correctly treated as not-done — but they accumulate. Sweep them at startup with for t in out_dir.glob("*.tmp"): t.unlink(), or delete each stale temp just before you rewrite it, as in Example 4.

Manifest and outputs drifting out of sync

If someone deletes an output file by hand but leaves its done row in the manifest, the job will skip it and the output stays missing. When correctness matters, verify on load — keep a file in done only if (out_dir / expected_name).exists() is also true — so a manually deleted output is automatically rescheduled.

The manifest file itself getting corrupted

A crash while rewriting a whole-file JSON manifest (Example 2) can truncate it, and then startup fails to parse it. An append-only CSV is far safer here: a crash can at worst leave one torn final line, which csv.DictReader skips, and every prior row is intact. Prefer append-only for long or unattended jobs, and back up the manifest before a risky retry pass.

Duplicate rows for the same file

Appending a new row per attempt means a retried file appears more than once in the manifest. This is intentional — it preserves the history — but every reader in this article resolves it by letting the last status for a file win. If you would rather keep the manifest unique, deduplicate on load by reading rows into a dict keyed on filename.

Changing the processing logic between runs

Resumability assumes "done" means done the way you want it now. If you change the transformation — a different target CRS, a new filter — the already-done files were produced by the old logic and will be skipped. When the logic changes materially, start a fresh manifest and output folder (or add a version tag to the manifest and treat rows from an older version as not-done) so everything is reprocessed under the new rules.

Combining with parallelism

A serial loop can append to one open manifest safely; multiple worker processes cannot, because concurrent appends interleave and corrupt rows. When you scale out with concurrent.futures.ProcessPoolExecutor, have each worker return its (file, status, detail) result and let the single parent process write the manifest as results arrive. See the parallel-processing guide below for the full pattern.

Internal links

How to Batch Process GIS Files in Python — the cornerstone guide this pattern extends.
How to Batch Process Multiple Shapefiles in Python — the base loop resumability is layered on top of.
How to Log and Summarise Errors in a Batch Job — turn the failed rows into a proper error report.
Speed Up Batch Jobs with Parallel Processing — run the remaining work-list across multiple cores.
How to Automate GIS Workflows with Python — schedule and wire resumable jobs into a larger pipeline.
How to Process Files in Nested Subfolders — build the work-list from a directory tree with rglob.

FAQ

What is the difference between a resumable job and one with error handling?

Error handling keeps a single run alive when one file fails — it catches the exception and continues. Resumability lets a new run pick up where a previous one stopped, so a crash, reboot, or manual stop does not cost you completed work. You want both: try/except per file to survive individual failures, and a manifest so restarting the whole job is cheap. The error-report guide covers the logging half in depth.

Should I use a CSV or JSON manifest?

Use append-only CSV for long or unattended jobs: appends are cheap, a crash can only damage the last line, and you can open it in any spreadsheet. Use JSON when you want structured per-file metadata and the batch is small enough that rewriting the whole file each iteration is not a concern. Both are shown above — CSV in Example 3, JSON in Example 2.

Why not just check whether the output file exists?

Because existence does not imply completeness. If the job died while writing that file, it exists but is truncated, and an existence check will skip it as if it were finished. The check only becomes safe once you write atomically — temp file then Path.replace() — so that the final name never appears until the write is complete.

How does `Path.replace()` make a write atomic?

Path.replace() calls the operating system's rename operation, which is atomic on a single filesystem: at every moment the destination path either holds the old file or the new one, never a partial mix. By doing all the slow, failure-prone writing on a .tmp and only renaming at the very end, you guarantee the real output name only ever points at a fully written file. Keep the .tmp in the same directory as the final file so the rename stays on one filesystem.

What does idempotent mean for a batch job?

It means running the job again produces the same result as running it once — no duplicated features, no double-appended rows, no corruption. You achieve it by deriving output names deterministically from inputs, replacing output instead of appending to it, and recording status per file. Idempotence is what makes "resume after a crash" simply a matter of running the script again.

How do I retry only the files that failed?

Read the manifest, select the rows whose status is failed, and run your processing over just that list, as in Example 5. Because each successful retry appends a new done row and readers take the latest status per file, a retried success cleanly supersedes the earlier failure without you having to edit the manifest by hand.

Can I combine this with parallel processing?

Yes, but do not let multiple workers write the manifest at once — concurrent appends corrupt it. Have each worker process return its result to the parent, and let the single parent process own the manifest and write rows as results come back. The work-list computation and atomic per-file writes are unchanged; only the recording moves to the parent. See Speed Up Batch Jobs with Parallel Processing.

What happens to half-written `.tmp` files if the job crashes?

They stay in the output folder as harmless orphans — the final output name was never created, so those inputs are correctly seen as not-done and get reprocessed. Clean them up by globbing *.tmp and unlinking at startup, or by deleting the specific temp just before you rewrite it, which also keeps the job idempotent.

How to Build a Resumable Batch GIS Job in Python #

Problem statement #

Quick answer #

Step-by-step solution #

Start with the simplest guard: skip if the output exists #

Record progress in a manifest #

Read the manifest at startup to compute the work-list #

Write each output atomically #

Append a status row as each file finishes #

Add a retry pass for the failures #

Code examples #

Example 1: The simplest resumable guard — skip if output exists #

Example 2: A JSON manifest of completed files, loaded to filter the work-list #

Example 3: Atomic write plus an append-only CSV manifest #

Example 4: Idempotent processing you can rerun safely #

Example 5: A retry-only pass over the failed entries #

Explanation #

Why "output exists" alone is fragile #

What atomic writes actually guarantee #

Why idempotence is the real goal #

Edge cases or notes #

Leftover .tmp files after a crash #

Manifest and outputs drifting out of sync #

The manifest file itself getting corrupted #

Duplicate rows for the same file #

Changing the processing logic between runs #

Combining with parallelism #

Internal links #

FAQ #

What is the difference between a resumable job and one with error handling? #

Should I use a CSV or JSON manifest? #

Why not just check whether the output file exists? #

How does Path.replace() make a write atomic? #

What does idempotent mean for a batch job? #

How do I retry only the files that failed? #

Can I combine this with parallel processing? #

What happens to half-written .tmp files if the job crashes? #