How to Batch Process Layers in QGIS with PyQGIS

QGIS has a batch mode built into every algorithm dialog, and it is genuinely useful right up to the moment you need a condition, a second step, or a report of what failed. Then you want a loop. The loop is not hard — for path in folder.glob("*.gpkg") around a processing.run call is most of it — but the difference between a script that survives a folder of real data and one that dies on file seven comes down to three habits: isolate each file's failure, mirror the input structure deliberately, and write a report at the end.

Problem statement

You have a folder of layers, or one GeoPackage with twenty of them, and one operation to apply to each. The obstacles:

The built-in batch dialog runs one algorithm. Your workflow is reproject, then fix geometries, then clip — three dialogs, three passes, intermediate files everywhere.
One bad file kills the run. File 7 of 200 has a null geometry, the exception propagates, and files 8–200 never ran.
No record of what happened. The run finished; you do not know which files were skipped, which produced zero features, or which took ten minutes.
Output naming is ad hoc. Results land in one flat folder with names that collide, or lose the subfolder structure that meant something.
Layers inside containers are invisible. A GeoPackage holding twelve layers looks like one file to a glob.
It is slow. Sequential, single-process, and the machine has eight cores idle.

The goal: one script, a folder in, a mirrored folder out, a CSV report, and a non-zero exit code if anything failed.

Quick answer

Wrap the algorithm in a loop and put the try inside the loop, not around it.

The five stages of a PyQGIS batch: discover inputs, load each layer, run the algorithm, write the output, append to the report. — Discover, load, run, write, record — the loop body is one file's whole life.

import csv
from pathlib import Path
import processing
from qgis.core import QgsVectorLayer

SRC, DST = Path("data/raw"), Path("data/processed")
DST.mkdir(parents=True, exist_ok=True)
rows = []

for path in sorted(SRC.glob("*.gpkg")):
    try:
        layer = QgsVectorLayer(str(path), path.stem, "ogr")
        if not layer.isValid():
            raise RuntimeError("layer failed to load")

        out = DST / f"{path.stem}_buffer.gpkg"
        processing.run("native:buffer", {
            "INPUT": layer, "DISTANCE": 25, "SEGMENTS": 8,
            "DISSOLVE": False, "OUTPUT": str(out),
        })
        rows.append({"file": path.name, "status": "ok", "features": layer.featureCount()})
    except Exception as exc:
        rows.append({"file": path.name, "status": "failed", "error": str(exc)})

with open(DST / "report.csv", "w", newline="") as fh:
    w = csv.DictWriter(fh, fieldnames=["file", "status", "features", "error"])
    w.writeheader()
    w.writerows(rows)

That runs headlessly with the bootstrap from running PyQGIS headless, and the algorithm call is the one from running Processing algorithms from Python.

Step-by-step solution

Discover the inputs, including the ones inside containers

A folder of shapefiles is easy. A folder of GeoPackages is not, because each one may hold several layers.

from pathlib import Path
from osgeo import ogr

def discover(folder: Path):
    """Yield (source_uri, label) for every vector layer under `folder`."""
    for path in sorted(folder.rglob("*")):
        if path.suffix.lower() in {".shp", ".geojson", ".fgb"}:
            yield str(path), path.stem
        elif path.suffix.lower() in {".gpkg", ".sqlite"}:
            ds = ogr.Open(str(path))
            if ds is None:
                continue
            for i in range(ds.GetLayerCount()):
                name = ds.GetLayerByIndex(i).GetName()
                yield f"{path}|layername={name}", f"{path.stem}__{name}"
            ds = None

Two details earn their place. rglob walks subfolders, which is what you want once the archive grows — the mirroring pattern from processing files in nested subfolders applies unchanged. And the label combines container and layer name, so two layers called roads in different GeoPackages do not overwrite each other.

If you would rather not add a GDAL import, QGIS can enumerate sublayers itself:

from qgis.core import QgsProviderRegistry

parts = QgsProviderRegistry.instance().querySublayers("data/raw/city.gpkg")
for p in parts:
    print(p.name(), p.uri(), p.featureCount())

Put the failure boundary around one file

One try/except around the whole loop versus one per file — the first loses everything after the failure, the second loses only that file. — Where the `try` sits decides whether a bad file costs you one row or the whole night.

The rule is simple and constantly broken: one try per item, inside the loop. A try around the loop turns a single malformed geometry into a total loss. Inside the loop, that file records failed, the loop continues, and you have 199 results plus one clear error to investigate.

Catch broadly here — except Exception — because the failure modes are genuinely varied: a QgsProcessingException from the algorithm, an OSError from a locked file, a RuntimeError you raised for an invalid layer. What matters is that the loop survives and the reason is recorded.

Make the output path a pure function of the input path

Naming is where batch jobs quietly corrupt themselves. Derive the destination mechanically so re-running produces the same layout:

def output_for(src: Path, root: Path, dst_root: Path, suffix: str) -> Path:
    rel = src.relative_to(root).with_suffix("")
    out = dst_root / rel.parent / f"{rel.name}_{suffix}.gpkg"
    out.parent.mkdir(parents=True, exist_ok=True)
    return out

data/raw/north/parcels.shp becomes data/processed/north/parcels_buffer.gpkg. The subfolder survives, the extension is normalised to GeoPackage, and nothing can collide unless the inputs already did.

Skip work that is already done

Any batch you run more than once should be idempotent and cheap on the second pass:

if out.exists() and out.stat().st_mtime > src.stat().st_mtime:
    rows.append({"file": src.name, "status": "skipped"})
    continue

That is the smallest possible version of the caching idea: if the output is newer than the input, there is nothing to do. For a run that may be interrupted, the fuller treatment in building a resumable batch job records completion in a manifest instead of trusting timestamps.

Chain several algorithms per file

The reason to write a loop instead of using the batch dialog is that the loop body can be a whole workflow:

def process_one(uri: str, out_path: Path) -> dict:
    layer = QgsVectorLayer(uri, "in", "ogr")
    if not layer.isValid():
        raise RuntimeError(f"invalid layer: {uri}")

    stats = {"features_in": layer.featureCount(), "crs_in": layer.crs().authid()}

    reprojected = processing.run("native:reprojectlayer", {
        "INPUT": layer, "TARGET_CRS": "EPSG:27700", "OUTPUT": "TEMPORARY_OUTPUT",
    })["OUTPUT"]

    fixed = processing.run("native:fixgeometries", {
        "INPUT": reprojected, "OUTPUT": "TEMPORARY_OUTPUT",
    })["OUTPUT"]

    result = processing.run("native:clip", {
        "INPUT": fixed, "OVERLAY": "data/district.geojson", "OUTPUT": str(out_path),
    })["OUTPUT"]

    written = QgsVectorLayer(str(out_path), "out", "ogr")
    stats["features_out"] = written.featureCount()
    return stats

Three algorithms, one file written, and the counts in and out returned so the report can flag layers that lost everything to the clip.

Report on every run

import csv

FIELDS = ["label", "status", "features_in", "features_out", "crs_in", "seconds", "error"]

def write_report(rows, path):
    with open(path, "w", newline="") as fh:
        w = csv.DictWriter(fh, fieldnames=FIELDS, extrasaction="ignore")
        w.writeheader()
        w.writerows(rows)

    failed = sum(1 for r in rows if r["status"] == "failed")
    empty = sum(1 for r in rows if r.get("features_out") == 0)
    print(f"{len(rows)} layers · {failed} failed · {empty} produced zero features")
    return failed

Return the failure count and use it as the exit code, so a scheduler notices. The empty-output count is the check that catches the silent disaster — every file "succeeded", every output has zero features, because the clip boundary was in the wrong CRS.

Code examples

Example 1: The complete batch script

#!/usr/bin/env python3
"""Batch: reproject → fix → clip every vector layer under data/raw."""
import csv
import logging
import sys
import time
from pathlib import Path

from qgis_headless import qgis_headless      # see the headless guide

SRC = Path("data/raw")
DST = Path("data/processed")
log = logging.getLogger("batch")

def main():
    import processing
    from qgis.core import QgsVectorLayer

    rows = []
    for uri, label in discover(SRC):
        started = time.perf_counter()
        row = {"label": label, "status": "ok"}
        out = DST / f"{label}.gpkg"
        out.parent.mkdir(parents=True, exist_ok=True)
        try:
            row.update(process_one(uri, out))
        except Exception as exc:
            row["status"] = "failed"
            row["error"] = f"{type(exc).__name__}: {exc}"
            log.warning("✗ %s — %s", label, exc)
        else:
            log.info("✓ %s (%d → %d features)", label, row["features_in"], row["features_out"])
        row["seconds"] = round(time.perf_counter() - started, 2)
        rows.append(row)

    return write_report(rows, DST / "report.csv")

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)-7s %(message)s")
    DST.mkdir(parents=True, exist_ok=True)
    with qgis_headless():
        failures = main()
    sys.exit(1 if failures else 0)

Example 2: Batching over layers inside one GeoPackage

Often the "folder" is a single container, and the loop is over sublayers.

from qgis.core import QgsProviderRegistry
import processing

SRC = "data/city.gpkg"

for sub in QgsProviderRegistry.instance().querySublayers(SRC):
    if sub.wkbType() == 0:          # skip non-spatial tables
        continue
    processing.run("native:buffer", {
        "INPUT": sub.uri(),
        "DISTANCE": 10, "SEGMENTS": 8, "DISSOLVE": False,
        "OUTPUT": f"ogr:dbname='out/city_buffers.gpkg' table=\"{sub.name()}\" (geom)",
    })

Writing every result back into a single output GeoPackage keeps the deliverable to one file, which is usually what the person receiving it wants.

Example 3: Per-file settings from a CSV

Not every layer wants the same parameters. Drive them from a table rather than an if ladder.

import csv
from pathlib import Path
import processing

settings = {}
with open("config/buffers.csv") as fh:
    for row in csv.DictReader(fh):
        settings[row["layer"]] = float(row["distance_m"])

DEFAULT = 25.0
for path in sorted(Path("data/raw").glob("*.gpkg")):
    distance = settings.get(path.stem, DEFAULT)
    processing.run("native:buffer", {
        "INPUT": str(path), "DISTANCE": distance, "SEGMENTS": 8,
        "DISSOLVE": False, "OUTPUT": f"data/processed/{path.stem}_buffer.gpkg",
    })

The CSV is reviewable, diffable, and editable by someone who does not write Python — the same argument that makes a YAML config worth the effort at pipeline scale.

Example 4: Parallelising across processes

QGIS objects are not thread-safe, so parallelism means processes, each with its own application.

from concurrent.futures import ProcessPoolExecutor
from pathlib import Path

def worker(uri_label):
    """Runs in a fresh process: start QGIS, do one layer, shut down."""
    from qgis_headless import qgis_headless
    uri, label = uri_label
    with qgis_headless():
        try:
            stats = process_one(uri, Path("data/processed") / f"{label}.gpkg")
            return {"label": label, "status": "ok", **stats}
        except Exception as exc:
            return {"label": label, "status": "failed", "error": str(exc)}

if __name__ == "__main__":
    items = list(discover(Path("data/raw")))
    with ProcessPoolExecutor(max_workers=4) as pool:
        rows = list(pool.map(worker, items))
    write_report(rows, Path("data/processed/report.csv"))

Starting a QGIS application per task costs a second or two, so this pays off for layers that take much longer than that and hurts for tiny ones. Chunk the work — give each worker a list of ten layers and start QGIS once per chunk — when the files are small. The trade-offs are the same as in speeding up batch GIS jobs with parallel processing.

Example 5: Using `qgis_process` with no Python at all

For a genuinely single-algorithm batch, the shell is enough:

for f in data/raw/*.gpkg; do
  qgis_process run native:buffer -- \
    INPUT="$f" DISTANCE=25 SEGMENTS=8 \
    OUTPUT="data/processed/$(basename "${f%.gpkg}")_buffer.gpkg" \
    || echo "FAILED: $f" >> data/processed/failures.txt
done

The || is the shell's version of per-item error isolation. This is a perfectly good answer for a one-off; move to Python when you need a second step, a condition, or a real report.

Explanation

A batch run report table with per-layer status, feature counts in and out, duration, and error text. — The zero-out row is the one the report exists to catch — it never raised an exception.

A batch script has two jobs, and only one of them is the GIS. The first is applying the operation; the second is accounting — knowing what was attempted, what succeeded, what changed, and what to look at. Scripts that skip the accounting feel finished and are not, because the failure mode they cannot see is the one that matters: a run where every file succeeded and every output is wrong.

That is why the report carries counts, not just statuses. A clip whose overlay is in the wrong CRS produces empty outputs and raises nothing at all — no exception, no warning, just twelve valid GeoPackages containing zero features. features_in and features_out side by side make it obvious in one glance. The same logic applies to a reprojection that silently assumed a CRS, or a join that matched nothing; the assertion you want is not "did it run" but "is the result the shape I expected", which is the argument behind validating a GeoDataFrame against a schema and applies equally here.

The loop-versus-batch-dialog choice is worth stating plainly, because the dialog is genuinely good. Use the built-in batch mode when the job is one algorithm, run once, with no conditions. Write a loop when you need more than one step, per-file parameters, error isolation, a report, resumability, or a schedule — that is, whenever it will run again. The moment the words "and then also" appear in the description, you want the loop.

The final structural point is about where QGIS fits at all. Much of what a batch script does — walking folders, building output paths, handling errors, writing CSV — is plain Python, identical to the general batch processing pattern used with GeoPandas. QGIS supplies the middle of the loop: the algorithms. That means you can develop and test the skeleton without QGIS running, and it means the choice between PyQGIS and GeoPandas for a given job is really a choice about which algorithms you need, not about which framework batches better. PyQGIS vs GeoPandas works through that decision.

Edge cases or notes

Mixed geometry types across a folder

A folder of "the same" layers often is not: some point, some multipoint, some with Z values. Algorithms that write a sink declare one geometry type, so a mixed batch can fail on file 40 with a type mismatch. Either branch on layer.wkbType(), or normalise first with native:promotetomulti and native:dropmzvalues.

Shapefile field-name truncation

Writing results to shapefile silently truncates field names to ten characters and can collide two columns into one. Write GeoPackage unless something downstream demands otherwise — the details are in shapefile column names truncated.

Locked outputs on Windows

A GeoPackage still open in a running QGIS Desktop cannot be overwritten by a script. The error is an unhelpful write failure. Close the layer in the desktop, or write to a temporary name and rename after — a rename is atomic and cheap.

Empty inputs are not errors

A layer with zero features is valid and most algorithms will happily produce an empty output. Decide explicitly whether that is a skipped or a failed in your report; the worst answer is ok.

Memory over a long run

One process working through a thousand layers accumulates provider caches. If resident memory climbs steadily, chunk the batch across process restarts rather than hunting for the leak — and see fixing memory errors with large files for the general shape of the problem.

Sorting matters for reproducibility

glob returns filesystem order, which differs between machines. Always sorted() the discovery so two runs process the same files in the same sequence, and a diff of two reports lines up.

Internal links

FAQ

Why write a loop when QGIS has a batch mode in every dialog?

Because the dialog runs exactly one algorithm with one set of parameters and reports nothing you can act on programmatically. A loop gives you multi-step workflows, per-file parameters, error isolation, a CSV report, resumability, and the ability to schedule the whole thing. Use the dialog for a one-off; write a loop for anything that will run twice.

How do I process every layer inside a GeoPackage?

Enumerate the sublayers rather than treating the file as one dataset. QgsProviderRegistry.instance().querySublayers(path) returns a part for each layer with its full URI, name, and geometry type; iterate those and pass sub.uri() as the algorithm input. The GDAL route — ogr.Open(path) and GetLayerByIndex — works equally well.

Where should the `try/except` go?

Inside the loop, wrapping the work for one file. Around the loop it converts a single bad file into total failure, which is the most common and most expensive mistake in batch GIS code. Catch Exception broadly, record the file and the error text in your report row, and let the loop continue.

How do I make the batch resumable after an interruption?

Skip any item whose output already exists and is newer than its input, or — more robustly — record completed items in a manifest file and consult it at startup. The manifest survives partially written outputs, which timestamps do not. The full pattern is in building a resumable batch GIS job.

Can I run PyQGIS batches in parallel threads?

No. QGIS objects and the Processing framework are not thread-safe, and threading them produces crashes rather than speedups. Parallelise with processes instead — ProcessPoolExecutor, one QgsApplication per worker — and chunk the work so the startup cost is amortised over several layers per process.

Why did every file succeed but produce zero features?

Almost always a CRS mismatch between the input layers and an overlay: a clip or intersection between layers in different coordinate systems finds no overlap and returns an empty result without raising. Record features_in and features_out in your report so this is visible, and reproject everything to a common CRS as the first step of the loop.

Should I write outputs as separate files or into one GeoPackage?

One GeoPackage with many layers is usually the better deliverable: a single file to move, atomic-ish to copy, no sidecar sprawl, no field-name truncation. Write separate files when downstream consumers expect them, or when parallel workers would otherwise contend for the same container — SQLite tolerates concurrent readers far better than concurrent writers.

How to Batch Process Layers in QGIS with PyQGIS #

Problem statement #

Quick answer #

Step-by-step solution #

Discover the inputs, including the ones inside containers #

Put the failure boundary around one file #

Make the output path a pure function of the input path #

Skip work that is already done #

Chain several algorithms per file #

Report on every run #

Code examples #

Example 1: The complete batch script #

Example 2: Batching over layers inside one GeoPackage #

Example 3: Per-file settings from a CSV #

Example 4: Parallelising across processes #

Example 5: Using qgis_process with no Python at all #

Explanation #

Edge cases or notes #

Mixed geometry types across a folder #

Shapefile field-name truncation #

Locked outputs on Windows #

Empty inputs are not errors #

Memory over a long run #

Sorting matters for reproducibility #

Internal links #

FAQ #

Why write a loop when QGIS has a batch mode in every dialog? #

How do I process every layer inside a GeoPackage? #

Where should the try/except go? #

How do I make the batch resumable after an interruption? #

Can I run PyQGIS batches in parallel threads? #

Why did every file succeed but produce zero features? #

Should I write outputs as separate files or into one GeoPackage? #