⚙️ How-To

How to Log and Summarise Errors in a Batch GIS Job in Python

You kick off a batch job over 400 shapefiles, walk away, and come back to a terminal that scrolled past its buffer an hour ago. Something failed — the run says so — but you have no idea which files, why, or how many. A wall of print() output is not a record; it is noise that disappears the moment the terminal closes. This guide shows you how to replace those scattered prints with real observability: Python's logging module writing to both console and a file, per-file exception handling so one bad file never aborts the run, and an end-of-run summary report that tells you exactly what happened.

Problem statement

You are running the same GeoPandas operation across a folder of files and hitting some mix of these problems:

  • Scattered print() calls — status messages, warnings, and errors all look identical, have no timestamps, and vanish when the shell closes, so you cannot reconstruct what happened.
  • One bad file kills the whole run — an unreadable shapefile or a corrupt geometry raises an exception that stops the loop, and the 380 files that would have processed fine never get a chance.
  • No traceback when it matters — you print str(exc) and get "Length mismatch" with no line number, no file name, and no stack, so you cannot tell where it actually broke.
  • No count at the end — the job finishes and you have no idea whether it processed 400 files or 12, how many failed, or how many you deliberately skipped.
  • No machine-readable failure list — to retry the broken files you have to eyeball terminal output and copy names by hand.

The goal: one repeatable script that logs every file to console and a persistent file, catches per-file exceptions without aborting, collects a structured result per file, and prints a summary report plus a CSV of failures you can act on.

Quick answer

Configure Python's logging module once with two handlers — a console handler and a file handler sharing one formatter — then use logging.getLogger(__name__) throughout. Wrap each file's work in try/except, call logger.exception() inside the except to capture the full traceback, and append a result dict (file, status, rows, error) per file. At the end, tally the statuses and write the failures to CSV with pandas.

A batch loop logging each file to console and a log file, then writing a summary and failures.csv.
One logger, two handlers: every file is logged to console and a file, then summarised with a CSV of failures.
import logging
import geopandas as gpd
import pandas as pd
from pathlib import Path

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)-8s %(message)s",
    handlers=[
        logging.StreamHandler(),                 # console
        logging.FileHandler("batch.log"),        # persistent file
    ],
)
logger = logging.getLogger(__name__)

src_dir = Path("data/raw")
results = []

for path in sorted(src_dir.glob("*.shp")):
    try:
        gdf = gpd.read_file(path)
        gdf = gdf.to_crs("EPSG:4326")
        gdf.to_file(f"data/clean/{path.stem}.gpkg", driver="GPKG")
        logger.info("ok    %s (%d rows)", path.name, len(gdf))
        results.append({"file": path.name, "status": "ok", "rows": len(gdf), "error": ""})
    except Exception as exc:
        logger.exception("failed %s", path.name)   # captures the traceback
        results.append({"file": path.name, "status": "failed", "rows": 0, "error": str(exc)})

report = pd.DataFrame(results)
counts = report["status"].value_counts().to_dict()
logger.info("SUMMARY: %s", counts)

failures = report[report["status"] == "failed"]
if not failures.empty:
    failures.to_csv("data/clean/failures.csv", index=False)

That is the whole job. The rest of this article explains each decision so you can adapt the logging setup and reporting to your own pipeline instead of running it blind.

Step-by-step solution

Configure logging once, at the top

Set up logging a single time, before the loop, and never again. The key idea is one logger, many handlers: a StreamHandler prints to the console so you watch progress live, and a FileHandler writes the same records to batch.log so you have a permanent record. A Formatter stamps every line with a timestamp and level, which print() will never give you for free.

import logging
from pathlib import Path

Path("logs").mkdir(exist_ok=True)

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

formatter = logging.Formatter(
    "%(asctime)s | %(levelname)-8s | %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)

console = logging.StreamHandler()
console.setLevel(logging.INFO)          # console shows INFO and up
console.setFormatter(formatter)

file_handler = logging.FileHandler("logs/batch.log", mode="w", encoding="utf-8")
file_handler.setLevel(logging.DEBUG)    # file captures everything
file_handler.setFormatter(formatter)

logger.addHandler(console)
logger.addHandler(file_handler)

Because each handler has its own level, the console stays readable (INFO and above) while the file keeps a verbose DEBUG trail you can dig into later. Configure this once and every logger.info(...) call anywhere in the script fans out to both destinations.

Loop over the folder and log each file

With the logger in place, walk the folder with pathlib and log the start and outcome of each file. Use the level that matches intent: logger.debug() for the noisy details, logger.info() for normal progress, logger.warning() for something odd but survivable, and logger.error() / logger.exception() for failures.

from pathlib import Path

src_dir = Path("data/raw")
files = sorted(src_dir.glob("*.shp"))
logger.info("Starting batch: %d files from %s", len(files), src_dir)

for path in files:
    logger.debug("Reading %s", path)
    # ... process the file ...
    logger.info("Processed %s", path.name)

Passing the values as arguments (logger.info("Processed %s", path.name)) rather than pre-formatting the string with an f-string lets logging skip the string work entirely when that level is disabled — a small habit that matters in tight loops.

Catch per-file exceptions so the run survives

Wrap the read-process-write cycle for each file in its own try/except. This is the single most important pattern in a batch job: a corrupt geometry, a missing sidecar, or an out-of-memory file should produce a logged failure and move on, not a stack trace that aborts the remaining files. Inside the except, call logger.exception() — it logs at ERROR level and appends the full traceback automatically.

for path in files:
    try:
        gdf = gpd.read_file(path)
        # ... transform and write ...
        logger.info("ok    %s", path.name)
    except Exception:
        logger.exception("failed %s", path.name)   # full traceback in the log
        continue

Catching bare Exception here is deliberate: in a batch you want to survive any per-file failure, not just the ones you anticipated. The traceback lands in batch.log so you can diagnose it later without stopping the run.

Collect a structured result per file

Logging tells the story; a list of result dicts gives you the data. Append one record per file with a consistent shape — file, status, rows, error — regardless of outcome. This structured list is what powers the summary and the failures CSV, and keeping the keys identical for ok, failed, and skipped files means you can drop the whole thing straight into a DataFrame.

results = []

for path in files:
    try:
        gdf = gpd.read_file(path)
        if gdf.crs is None:
            logger.warning("skip  %s: no CRS", path.name)
            results.append({"file": path.name, "status": "skipped", "rows": len(gdf), "error": "no CRS"})
            continue
        gdf = gdf.to_crs("EPSG:4326")
        gdf.to_file(f"data/clean/{path.stem}.gpkg", driver="GPKG")
        results.append({"file": path.name, "status": "ok", "rows": len(gdf), "error": ""})
    except Exception as exc:
        logger.exception("failed %s", path.name)
        results.append({"file": path.name, "status": "failed", "rows": 0, "error": str(exc)})

Note the three distinct statuses: ok for success, failed for an exception, and skipped for a file you consciously chose not to process (here, a missing CRS). Keeping skipped separate from failed matters — a skip is a decision, a failure is a surprise.

Summarise counts and write a failures report

After the loop, turn the result list into a DataFrame, tally the statuses, log a one-line summary, and write just the failures to CSV. This is your end-of-run report: a glance tells you the health of the batch, and failures.csv gives you a machine-readable list to feed into a retry.

import pandas as pd

report = pd.DataFrame(results)
counts = report["status"].value_counts()

logger.info(
    "SUMMARY: %d ok, %d failed, %d skipped (of %d)",
    counts.get("ok", 0), counts.get("failed", 0),
    counts.get("skipped", 0), len(report),
)

failures = report[report["status"] == "failed"]
if not failures.empty:
    failures.to_csv("logs/failures.csv", index=False)
    logger.warning("Wrote %d failures to logs/failures.csv", len(failures))

The if not failures.empty guard means you only produce a failures file when there is something to report — a clean run leaves no misleading empty CSV behind.

Code examples

Example 1: A reusable logging setup function

Factor the configuration into a function you can call from any script. Guarding against duplicate handlers keeps it safe to call more than once (for example in a notebook, where re-running a cell would otherwise stack handlers and print every line several times).

import logging
from pathlib import Path

def setup_logging(log_path="logs/batch.log", level=logging.INFO):
    Path(log_path).parent.mkdir(parents=True, exist_ok=True)
    logger = logging.getLogger("batch")
    logger.setLevel(logging.DEBUG)

    if logger.handlers:            # already configured, don't stack handlers
        return logger

    fmt = logging.Formatter("%(asctime)s | %(levelname)-8s | %(message)s")

    console = logging.StreamHandler()
    console.setLevel(level)
    console.setFormatter(fmt)

    file_handler = logging.FileHandler(log_path, mode="w", encoding="utf-8")
    file_handler.setLevel(logging.DEBUG)
    file_handler.setFormatter(fmt)

    logger.addHandler(console)
    logger.addHandler(file_handler)
    return logger

logger = setup_logging()
logger.info("Logging ready")

Example 2: The batch loop with logger.exception

The full loop, wiring the logger, the per-file try/except, and the structured results together.

import geopandas as gpd
from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/clean")
out_dir.mkdir(parents=True, exist_ok=True)

results = []
for path in sorted(src_dir.glob("*.shp")):
    try:
        gdf = gpd.read_file(path)
        gdf = gdf.to_crs("EPSG:4326")
        out_path = out_dir / f"{path.stem}.gpkg"
        gdf.to_file(out_path, driver="GPKG")
        logger.info("ok    %s (%d rows)", path.name, len(gdf))
        results.append({"file": path.name, "status": "ok", "rows": len(gdf), "error": ""})
    except Exception as exc:
        logger.exception("failed %s", path.name)
        results.append({"file": path.name, "status": "failed", "rows": 0, "error": str(exc)})

Example 3: Building the summary and failures CSV with pandas

Turn the collected results into a report. value_counts() gives the tally; a boolean mask isolates the failures for the CSV.

import pandas as pd

report = pd.DataFrame(results)

summary = report["status"].value_counts().rename_axis("status").reset_index(name="count")
logger.info("Run summary:\n%s", summary.to_string(index=False))

failures = report.loc[report["status"] == "failed", ["file", "error"]]
if not failures.empty:
    failures.to_csv("logs/failures.csv", index=False)

print(f"Done: {len(report)} files, {len(failures)} failed")

Example 4: A context manager that times and logs each file

A small @contextmanager wraps every file in timing, success logging, and failure logging, so the loop body stays focused on the GIS work. It re-raises after logging, letting the caller record the failure in results.

import time
import logging
from contextlib import contextmanager

logger = logging.getLogger("batch")

@contextmanager
def logged_file(name):
    start = time.perf_counter()
    logger.debug("start %s", name)
    try:
        yield
    except Exception:
        elapsed = time.perf_counter() - start
        logger.exception("failed %s after %.2fs", name, elapsed)
        raise                       # let the loop record it as a failure
    else:
        elapsed = time.perf_counter() - start
        logger.info("ok    %s in %.2fs", name, elapsed)

# usage
for path in sorted(src_dir.glob("*.shp")):
    try:
        with logged_file(path.name):
            gdf = gpd.read_file(path).to_crs("EPSG:4326")
            gdf.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")
        results.append({"file": path.name, "status": "ok", "rows": len(gdf), "error": ""})
    except Exception as exc:
        results.append({"file": path.name, "status": "failed", "rows": 0, "error": str(exc)})

Example 5: A rotating file handler for long-running or repeated jobs

If you run the batch on a schedule, a plain FileHandler in append mode grows without bound. RotatingFileHandler caps each log file and keeps a fixed number of backups, so batch.log never eats your disk.

import logging
from logging.handlers import RotatingFileHandler

logger = logging.getLogger("batch")
logger.setLevel(logging.DEBUG)

handler = RotatingFileHandler(
    "logs/batch.log",
    maxBytes=2_000_000,      # rotate at ~2 MB
    backupCount=5,           # keep batch.log.1 ... batch.log.5
    encoding="utf-8",
)
handler.setFormatter(logging.Formatter("%(asctime)s | %(levelname)-8s | %(message)s"))
logger.addHandler(handler)

Explanation

Why logging beats print()

print() is a blunt instrument: every message looks the same, goes to one place, and disappears when the terminal does. The logging module solves four problems at once.

  • Levels. Every message carries a severity — DEBUG, INFO, WARNING, ERROR, CRITICAL. You can turn the volume down to WARNING for a quiet production run, or up to DEBUG when hunting a bug, without touching a single log call.
  • Timestamps and context. A Formatter stamps each line with the time, level, and (if you want) module name. When a batch failed at 03:14, you know when and where, not just what.
  • Multiple destinations. One logger fans out to as many handlers as you like — console for a live view, a file for the record, and a rotating file for scheduled runs — all from a single set of log calls.
  • One place to configure. Behaviour is set once at startup. With print(), changing where output goes means editing every call site; with logging, you change one handler.

The practical upgrade path is mechanical: every print("...") that reports progress becomes logger.info("..."), every warning becomes logger.warning("..."), and every error inside an except becomes logger.exception("...").

Why logger.exception() inside except is special

logger.exception() is the reason a logged batch is easier to debug than a printed one. Called from inside an except block, it does two things: it logs your message at ERROR level, and it automatically appends the full traceback of the exception currently being handled. You get the file name you passed in and the exact line and call chain that raised — the same detail Python would print on an uncaught crash, but captured neatly in batch.log while the run keeps going.

Compare it to print(f"failed: {exc}"), which gives you only the exception's message — often something uninformative like "Length mismatch: Expected axis has 3 elements" with no hint of where it happened. logger.exception() is equivalent to logger.error(msg, exc_info=True); use it whenever you are inside an except and want the traceback. Outside an exception handler, use logger.error() for failures you detect yourself.

Edge cases or notes

Duplicate log lines from stacked handlers

If every message prints two or three times, you have added handlers more than once — common when re-running a script in a notebook or Jupyter kernel that keeps the logger alive between runs. Guard your setup with if logger.handlers: return, as in Example 1, or call logger.handlers.clear() before adding fresh ones.

The root logger and basicConfig

logging.basicConfig(...) configures the root logger and is fine for a single script — it is the quickest way to get file-plus-console output, as in the Quick answer. But it only acts once (later calls are ignored unless you pass force=True), and it configures the root logger globally. For anything reusable, prefer a named logger via logging.getLogger(__name__) with handlers you attach yourself, so your logging config does not leak into imported libraries.

Keep skipped separate from failed

A file you deliberately skip (no CRS, empty geometry, already processed) is not a failure — it is a decision. Log skips at WARNING and give them their own status so your summary reads 380 ok, 4 failed, 16 skipped rather than lumping 20 files together as broken. When you later triage failures.csv, you want it to contain only genuine surprises.

Logging is not retrying

This article is about observability — knowing what happened. It does not restart failed files or resume an interrupted run. failures.csv is the handoff point: it is the input to a retry pass. For the mechanics of resuming a job, skipping already-completed files, and re-running only the failures, see the resumable-batch guide linked below.

Log the file path, not just the name

In a summary it is tempting to log path.name, but two files called roads.shp in different subfolders become indistinguishable. Log the full path (or a path relative to the source root) at DEBUG so you can always trace a failure back to the exact file, while keeping the human-friendly path.name in INFO lines.

Flushing and encoding on abrupt termination

FileHandler flushes on each record by default, so batch.log is usually current even if the process is killed. Still, pass encoding="utf-8" explicitly so file paths or error messages with non-ASCII characters do not raise a secondary error while logging, and call logging.shutdown() at the very end of a long job to guarantee every buffer is flushed and files are closed cleanly.

FAQ

Why should I use logging instead of print() in a batch job?

logging gives you severity levels, timestamps, and multiple output destinations from one configuration. You can send INFO progress to the console while writing a full DEBUG trail to a file, turn the verbosity up or down without editing call sites, and — critically — capture tracebacks with logger.exception(). print() offers none of that: one stream, no levels, no timestamps, and output that vanishes when the terminal closes.

How do I log to both the console and a file at the same time?

Attach two handlers to one logger: a StreamHandler() for the console and a FileHandler("batch.log") for the file. Give them a shared Formatter so both destinations look consistent, and set a level per handler if you want the console quieter than the file. Every logger.info(...) call then writes to both. The Quick answer shows the minimal version via basicConfig; Example 1 shows the reusable function form.

What is the difference between logger.error() and logger.exception()?

Both log at ERROR level, but logger.exception() additionally attaches the current traceback — so it only makes sense inside an except block. Use logger.exception("failed %s", name) when handling a caught exception to get the full stack in your log. Use logger.error(...) for error conditions you detect yourself, where there is no active exception to capture.

How do I stop one bad file from aborting the whole batch?

Wrap each file's read-process-write cycle in its own try/except, log the problem with logger.exception(), record a failed result, and continue to the next file. Catch broad Exception here on purpose — in an unattended batch you want to survive any per-file error and deal with it afterward from the summary, rather than lose the remaining files to a single crash.

How do I produce a summary of how many files succeeded or failed?

Append a result dict with a status field (ok, failed, or skipped) for every file, build a DataFrame from the list, and call report["status"].value_counts(). Log that as a one-line summary at the end of the run. Because each record has the same keys, the tally is a single pandas call regardless of how the file turned out.

How do I export the list of failed files to a CSV?

Collect an error field alongside the status, then filter the report DataFrame to the failed rows and call .to_csv("failures.csv", index=False). Guard it with if not failures.empty so a clean run does not leave an empty file behind. The CSV of file names and error messages is exactly what a retry pass needs as input.

Why do my log messages appear multiple times?

You have added handlers to the same logger more than once — usually by re-running a script in a notebook or calling your setup function repeatedly. Each duplicate handler emits its own copy of every record. Guard the setup with if logger.handlers: return logger, or clear existing handlers with logger.handlers.clear() before adding new ones.

Should logging handle retrying failed files too?

No — keep the two concerns separate. Logging and the summary tell you what happened; retrying is a distinct step that reads failures.csv and reprocesses only those files. Mixing retry logic into the logging loop makes both harder to reason about. See How to Build a Resumable Batch GIS Job in Python for the retry and resume mechanics.