How to Batch Process a Folder of GIS Files in Python: The Complete Workflow

Sooner or later every GIS project stops being about one file. A colleague drops forty shapefiles into a shared drive, a portal exports a folder of GeoJSON per district, and suddenly the tidy thing you did once by hand needs to happen a hundred times, identically, without you babysitting it. The good news is that almost every batch job in Python is the same shape underneath: discover the files, read each one, run a single reusable transformation, write the result somewhere safe, and keep a record of what happened. This article teaches that pattern end to end so you can drop your own logic into the middle and trust the rest.

Problem statement

You have a folder of vector data and you need to do the same thing to every file in it. You are hitting some mix of these:

The same manual steps, over and over — you open each file in QGIS, reproject it, add a field, export it, and repeat, and it is both slow and error-prone.
A mix of formats in one folder — some inputs are .shp, some .geojson, some .gpkg, and you want one script that handles all of them.
One bad file kills the run — a corrupt geometry or a missing sidecar throws halfway through and you lose the work already done.
No idea what actually happened — after the batch finishes you cannot tell which files succeeded, which were skipped, and how many features came out.
Inputs and outputs getting tangled — processed files land back in the source folder and you can no longer tell clean data from raw.

The goal: one repeatable script that finds every file, applies a single processing function to each, writes clean output to a separate folder, survives bad files, and hands you a summary of exactly what it did.

Quick answer

Use pathlib to glob the folder, loop over the files, pass each GeoDataFrame through one reusable process_gdf() function, and write the result to a separate output folder. Wrap every file in its own try/except so one failure never stops the batch, and collect a row of metadata per file for a summary report at the end.

A five-stage pipeline: discover files, read each one, apply one reusable process, write output, and report. — Every batch job is a file loop wrapped around one reusable transformation.

import geopandas as gpd
import pandas as pd
from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)

PATTERNS = ("*.shp", "*.geojson", "*.gpkg")   # formats to accept

def process_gdf(gdf):
    """The one reusable transformation. Edit this; leave the loop alone."""
    gdf = gdf.to_crs("EPSG:4326")
    gdf["area_m2"] = gdf.to_crs("EPSG:3857").area
    return gdf

files = sorted(p for pat in PATTERNS for p in src_dir.glob(pat))

summary = []
for path in files:
    try:
        gdf = gpd.read_file(path)
        result = process_gdf(gdf)
        out_path = out_dir / f"{path.stem}.gpkg"
        result.to_file(out_path, driver="GPKG")
        summary.append({"file": path.name, "rows": len(result), "status": "ok"})
    except Exception as exc:
        summary.append({"file": path.name, "rows": 0, "status": f"error: {exc}"})

report = pd.DataFrame(summary)
print(report.to_string(index=False))
report.to_csv(out_dir / "_run_report.csv", index=False)

That is the whole workflow. The rest of this article unpacks each stage so you can swap in your own processing safely instead of running it blind.

Step-by-step solution

Keep inputs and outputs separate

Before any code, decide on two folders: one for source data you never touch, one for results. Writing processed files back into the source folder is how you lose the ability to re-run a batch, because your second run reads its own output as input.

from pathlib import Path

src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)

mkdir(parents=True, exist_ok=True) creates the output folder (and any missing parents) and does nothing if it already exists, so it is safe to run every time.

Discover the files

Use pathlib.Path plus glob to find the files. Globbing one pattern at a time and chaining the results keeps things readable and lets you control exactly which extensions you accept. Sorting the result makes runs deterministic and the logs easy to read.

from pathlib import Path

src_dir = Path("data/raw")
PATTERNS = ("*.shp", "*.geojson", "*.gpkg")

files = sorted(p for pat in PATTERNS for p in src_dir.glob(pat))
print(f"Found {len(files)} files")
for f in files:
    print(f.name)

Globbing *.shp deliberately ignores the .dbf, .shx, and .prj sidecars — GeoPandas reads them automatically from the .shp. If your data is nested in subfolders, swap glob for rglob; there is a dedicated guide on processing files in nested subfolders when that gets involved.

Read each file

Loop over the paths and load each into a GeoDataFrame with gpd.read_file(). The same call reads shapefiles, GeoJSON, GeoPackage, and every other format GDAL supports, which is what lets one loop handle a mixed folder.

import geopandas as gpd

for path in files:
    gdf = gpd.read_file(path)
    print(f"{path.name}: {len(gdf)} features, CRS {gdf.crs}")

At this point each gdf is an ordinary GeoDataFrame, so anything you would do to a single file works here too.

Apply one reusable processing function

This is the heart of the pattern. Put the repeated GIS logic in a single function that takes a GeoDataFrame and returns a GeoDataFrame. The loop stays fixed; you only ever edit this function.

def process_gdf(gdf):
    gdf = gdf.copy()

    if gdf.crs is None:
        raise ValueError("Input has no CRS; cannot reproject safely")

    gdf = gdf.to_crs("EPSG:4326")          # standardise the CRS
    gdf = gdf[gdf.geometry.notna()]        # drop rows with no geometry
    gdf["processed"] = True                # tag the output
    return gdf

Keeping the transformation separate from the loop has a real payoff: you can test process_gdf() on one file until it is right, then run it across the whole folder unchanged. The same skeleton handles reprojecting, filtering, adding columns, clipping to a boundary, or standardising a schema — only the body changes.

Write to the output folder

Write each result to the output folder with a name you can trace back to the source. GeoPackage is the recommended output: one file per dataset, the CRS stored internally, no field-name limits.

for path in files:
    gdf = gpd.read_file(path)
    result = process_gdf(gdf)
    out_path = out_dir / f"{path.stem}.gpkg"
    result.to_file(out_path, driver="GPKG")
    print(f"Saved {out_path.name}")

Using path.stem keeps the original base name and just changes the extension, so roads.shp becomes roads.gpkg. See choosing an output format below for when a different format makes sense.

Handle errors per file and report

Wrap the read-process-write cycle for each file in its own try/except so a single bad file is logged and skipped rather than aborting the run. Collect one record per file and turn it into a report at the end.

import pandas as pd

summary = []
for path in files:
    try:
        gdf = gpd.read_file(path)
        result = process_gdf(gdf)
        result.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")
        summary.append({"file": path.name, "rows": len(result), "status": "ok"})
    except Exception as exc:
        summary.append({"file": path.name, "rows": 0, "status": f"error: {exc}"})

report = pd.DataFrame(summary)
report.to_csv(out_dir / "_run_report.csv", index=False)
print(report.to_string(index=False))

That summary is your audit trail. For richer, structured logging across a big job, see log and summarise errors in a batch job.

Code examples

Example 1: The minimal discover-and-loop

The smallest thing that works: find every shapefile, read it, write a copy to the output folder. Start here and add logic to the middle.

from pathlib import Path
import geopandas as gpd

src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)

for path in sorted(src_dir.glob("*.shp")):
    gdf = gpd.read_file(path)
    gdf.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")
    print(f"Wrote {path.stem}.gpkg")

Example 2: A reusable process_gdf() you can swap out

Isolating the transformation is the single most useful habit in batch GIS. The loop below never changes no matter what process_gdf() does.

from pathlib import Path
import geopandas as gpd

src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)

def process_gdf(gdf):
    """Reproject, keep only valid polygons, add an area column."""
    gdf = gdf.copy()
    if gdf.crs is None:
        raise ValueError("Missing CRS")
    gdf = gdf.to_crs("EPSG:27700")            # metres, for area
    gdf = gdf[gdf.geometry.is_valid]
    gdf["area_m2"] = gdf.geometry.area
    return gdf

for path in sorted(src_dir.glob("*.shp")):
    result = process_gdf(gpd.read_file(path))
    result.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")

Example 3: Support multiple input formats at once

Real folders are rarely one format. Chain several glob patterns so a single run picks up shapefiles, GeoJSON, and GeoPackages together.

from pathlib import Path
import geopandas as gpd

src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)

PATTERNS = ("*.shp", "*.geojson", "*.json", "*.gpkg")

files = sorted(p for pat in PATTERNS for p in src_dir.glob(pat))
print(f"Found {len(files)} files across {len(PATTERNS)} patterns")

for path in files:
    gdf = gpd.read_file(path)
    gdf = gdf.to_crs("EPSG:4326")
    gdf.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")

Because gpd.read_file() detects the format from the file itself, the body of the loop does not care whether it is reading a shapefile or a GeoJSON.

Example 4: try/except with a collected summary list

The production shape: every file is wrapped, failures are recorded with the exception message, and the loop always finishes.

from pathlib import Path
import geopandas as gpd

src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)

def process_gdf(gdf):
    if gdf.crs is None:
        raise ValueError("Missing CRS")
    return gdf.to_crs("EPSG:4326")

summary = []
for path in sorted(src_dir.glob("*.shp")):
    print(f"Processing {path.name} ...")
    try:
        result = process_gdf(gpd.read_file(path))
        result.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")
        summary.append((path.name, len(result), "ok"))
    except Exception as exc:
        summary.append((path.name, 0, f"error: {exc}"))

ok = sum(1 for _, _, s in summary if s == "ok")
print(f"\nDone: {ok}/{len(summary)} succeeded")

Example 5: Write a CSV run-report with pandas

Turn the collected records into a DataFrame and save a CSV alongside the outputs. This is the artefact you keep — it says what ran, when, and how it went.

from pathlib import Path
from datetime import datetime
import geopandas as gpd
import pandas as pd

src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)

rows = []
for path in sorted(src_dir.glob("*.shp")):
    try:
        gdf = gpd.read_file(path)
        out = gdf.to_crs("EPSG:4326")
        out.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")
        rows.append({"file": path.name, "features": len(out),
                     "crs_in": str(gdf.crs), "status": "ok"})
    except Exception as exc:
        rows.append({"file": path.name, "features": 0,
                     "crs_in": "-", "status": f"error: {exc}"})

report = pd.DataFrame(rows)
report["run_at"] = datetime.now().isoformat(timespec="seconds")
report.to_csv(out_dir / "_run_report.csv", index=False)
print(report.to_string(index=False))

Explanation

This pattern works because nearly every folder-based GIS job decomposes into the same four moving parts: a way to find the files, a loop over them, a function that transforms one GeoDataFrame, and an output folder for the results. Once you see a batch job that way, the loop becomes boilerplate you can reuse forever, and all your thinking goes into the one function in the middle.

Separating the transformation from the loop is what makes the whole thing testable. You develop process_gdf() against a single representative file, in a notebook or the REPL, until it does exactly what you want. Only then do you point the loop at the whole folder. If something breaks later, you already know the loop is sound, so the bug is in the function — a much smaller place to look.

The per-file try/except matters more than it looks. Real folders always contain at least one surprise: an empty file, a broken geometry, a shapefile that lost its .prj. Without the guard, that one file throws an exception that unwinds the entire loop and you lose every result computed before it. With the guard, the bad file becomes a single row in your report and the batch runs to completion. The summary list then does double duty — it is both your progress log and the record you keep for later.

GeoPandas is the right tool when you have a moderate number of vector files and standard operations. When the folder gets large enough that runtime hurts, reach for parallel processing; when a job is long enough that you need it to survive an interruption, make it resumable. Those are refinements of this same skeleton, not replacements for it.

Edge cases or notes

Files with a missing or wrong CRS

If gdf.crs is None, calling to_crs() raises an error because there is no source projection to transform from. You must first attach the correct CRS with set_crs() — and you need to actually know what it is. Standardising a mixed folder before processing is a job in its own right; work through the Python GIS data cleaning checklist to catch missing CRS, invalid geometries, and schema drift up front.

Mixed geometry types in one folder

A folder can hold points, lines, and polygons. Some operations only make sense for one type — polygon area is meaningless for points, and writing mixed types into a single layer can fail. If your process_gdf() assumes polygons, guard it with a check on gdf.geom_type and skip or branch on anything unexpected.

Choosing the output format

Prefer GeoPackage (driver="GPKG") for output. Shapefiles split one dataset across four-plus files, truncate field names to 10 characters, cap at 2 GB, and store the CRS in a fragile .prj sidecar. GeoPackage is a single SQLite file with no field-name limits, an internal CRS, and support for many layers. Use GeoJSON only when a downstream tool specifically needs it. If you need several formats from one run, see batch export to multiple formats.

One output file per input, or one combined file

The examples here write one output per input, which keeps a clear trace back to the source. If your goal is instead to stitch everything into a single dataset, that is a merge, not a per-file write — collect the frames and concatenate them. See merge many shapefiles into one file for the correct CRS-safe way to do it.

Large folders and memory

gpd.read_file() loads a whole file into memory. Processing one file at a time inside the loop (as shown) keeps only one GeoDataFrame resident, so even a big folder stays manageable as long as no single file is enormous. If an individual file is too large to fit, read it in chunks with pyogrio options rather than loading it whole.

Rasters are a different pipeline

This workflow is for vector data read through GeoPandas. Rasters need Rasterio and a windowed read/write pattern, not read_file() and to_file(). If your folder is full of GeoTIFFs, go straight to batch process rasters with Rasterio.

Internal links

FAQ

How do I loop through every GIS file in a folder in Python?

Use pathlib with glob. For a single format, Path("data/raw").glob("*.shp") yields every shapefile. For several formats, chain patterns: sorted(p for pat in ("*.shp", "*.geojson", "*.gpkg") for p in src_dir.glob(pat)). Sorting makes the run order deterministic, which keeps logs readable and results reproducible.

Should I put my processing logic in a function?

Yes. Keeping the transformation in a single process_gdf(gdf) function separates the what (your GIS logic) from the how (the file loop). You can test the function on one file until it is correct, then run it across the folder without changing the loop. It also makes the code far easier to read and to reuse in the next batch job.

How do I stop one bad file from killing the whole batch?

Wrap the read-process-write cycle for each file in its own try/except, log the exception, and continue the loop. The failing file ends up as a single row in your summary marked as an error while every other file is still processed. Never let one corrupt geometry abort an overnight run.

Which output format should I use for a batch job?

GeoPackage is the best default: a single file, an internal CRS, no field-name limits, and support for multiple layers. Write it with result.to_file(path, driver="GPKG"). Reach for shapefile only when a legacy tool demands it, and GeoJSON only for web interchange. When you need several formats from the same run, use the multiple formats guide.

Can I mix shapefiles, GeoJSON, and GeoPackage in one run?

Yes. Chain multiple glob patterns to discover them and let gpd.read_file() detect each format automatically. The body of your loop does not need to know or care which format it is reading, because every file becomes an ordinary GeoDataFrame after the read.

How do I handle files that have no CRS?

to_crs() needs a source CRS, so a file where gdf.crs is None will raise. Attach the correct CRS first with set_crs() — but only when you actually know what it should be. Cleaning a whole mixed folder before you process it is a task of its own; the Python GIS data cleaning checklist walks through CRS repair alongside the other pre-batch checks.

When should I use the specialised batch guides instead of this one?

This article is the general pattern. Move to a specialised guide when your job has a specific shape: parallel processing for speed on big folders, a resumable job for long runs that must survive interruption, nested subfolders for tree-structured data, merging to combine into one file, or Rasterio for raster data. Each is a variation on the loop taught here.

How do I keep a record of what the batch did?

Collect one dictionary per file inside the loop — file name, feature count, status — then build a pandas.DataFrame and write it to CSV alongside your outputs. That report tells you which files succeeded, which failed and why, and how many features each produced. For structured, timestamped logging on larger jobs, see log and summarise errors in a batch job.

How to Batch Process a Folder of GIS Files in Python: The Complete Workflow #

Problem statement #

Quick answer #

Step-by-step solution #

Keep inputs and outputs separate #

Discover the files #

Read each file #

Apply one reusable processing function #

Write to the output folder #

Handle errors per file and report #

Code examples #

Example 1: The minimal discover-and-loop #

Example 2: A reusable process_gdf() you can swap out #

Example 3: Support multiple input formats at once #

Example 4: try/except with a collected summary list #

Example 5: Write a CSV run-report with pandas #

Explanation #

Edge cases or notes #

Files with a missing or wrong CRS #

Mixed geometry types in one folder #

Choosing the output format #

One output file per input, or one combined file #

Large folders and memory #

Rasters are a different pipeline #

Internal links #

FAQ #

How do I loop through every GIS file in a folder in Python? #

Should I put my processing logic in a function? #

How do I stop one bad file from killing the whole batch? #

Which output format should I use for a batch job? #

Can I mix shapefiles, GeoJSON, and GeoPackage in one run? #

How do I handle files that have no CRS? #

When should I use the specialised batch guides instead of this one? #

How do I keep a record of what the batch did? #