How to Batch Process a Folder of GIS Files in Python: The Complete Workflow
On this page
Sooner or later every GIS project stops being about one file. A colleague drops forty shapefiles into a shared drive, a portal exports a folder of GeoJSON per district, and suddenly the tidy thing you did once by hand needs to happen a hundred times, identically, without you babysitting it. The good news is that almost every batch job in Python is the same shape underneath: discover the files, read each one, run a single reusable transformation, write the result somewhere safe, and keep a record of what happened. This article teaches that pattern end to end so you can drop your own logic into the middle and trust the rest.
Problem statement
Doing the same thing to every file in a folder of vector data goes wrong in five familiar ways:
- The same manual steps, over and over — you open each file in QGIS, reproject it, add a field, export it, and repeat, and it is both slow and error-prone.
- A mix of formats in one folder — some inputs are
.shp, some.geojson, some.gpkg, and you want one script that handles all of them. - One bad file kills the run — a corrupt geometry or a missing sidecar throws halfway through and you lose the work already done.
- No idea what actually happened — after the batch finishes you cannot tell which files succeeded, which were skipped, and how many features came out.
- Inputs and outputs getting tangled — processed files land back in the source folder and you can no longer tell clean data from raw.
The answer is one processing function applied to every file the script finds, output to a separate folder, bad files survived and listed, and a summary at the end.
Quick answer
Use pathlib to glob the folder, loop over the files, pass each GeoDataFrame through one reusable process_gdf() function, and write the result to a separate output folder. Wrap every file in its own try/except so one failure never stops the batch, and collect a row of metadata per file for a summary report at the end.
import geopandas as gpd
import pandas as pd
from pathlib import Path
src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)
PATTERNS = ("*.shp", "*.geojson", "*.gpkg") # formats to accept
def process_gdf(gdf):
"""The one reusable transformation. Edit this; leave the loop alone."""
gdf = gdf.to_crs("EPSG:4326")
gdf["area_m2"] = gdf.to_crs("EPSG:8857").area # Equal Earth: an equal-area projection
return gdf
files = sorted(p for pat in PATTERNS for p in src_dir.glob(pat))
summary = []
for path in files:
try:
gdf = gpd.read_file(path)
result = process_gdf(gdf)
out_path = out_dir / f"{path.stem}_{path.suffix[1:]}.gpkg" # a.shp and a.geojson stay apart
result.to_file(out_path, driver="GPKG")
summary.append({"file": path.name, "rows": len(result), "status": "ok"})
except Exception as exc:
summary.append({"file": path.name, "rows": 0, "status": f"error: {exc}"})
report = pd.DataFrame(summary)
print(report.to_string(index=False))
report.to_csv(out_dir / "_run_report.csv", index=False) file rows status
lakes.shp 0 error: Cannot transform naive geometries. Please set a crs on the object first.
land.gpkg 127 ok
ne_110m_admin_0_boundary_lines_land.shp 331 ok
ne_110m_admin_0_countries.geojson 177 ok
ne_110m_admin_0_countries.shp 177 ok
ne_110m_admin_1_states_provinces.shp 51 ok
ne_110m_coastline.shp 134 ok
ne_110m_geography_regions_polys.shp 60 ok
ne_110m_glaciated_areas.shp 11 ok
ne_110m_lakes.shp 24 ok
ne_110m_land.shp 127 ok
ne_110m_ocean.shp 2 ok
ne_110m_populated_places.shp 243 ok
ne_110m_rivers_lake_centerlines.shp 13 ok
ne_50m_admin_0_countries.shp 242 ok
ne_50m_populated_places.shp 1251 ok
rivers.shp 13 ok
Three things in that report are worth a second look. The lakes delivered without a .prj fail, by name. The rivers file cut off mid-copy does not: it reads as 13 rows, 6 of them with no geometry, and goes through as ok. And the output names carry the input's extension, because this folder holds the countries twice — as a shapefile and as GeoJSON — and f"{path.stem}.gpkg" would have written both to ne_110m_admin_0_countries.gpkg, the second silently replacing the first. The area is computed in Equal Earth because an earlier version of this snippet used Web Mercator:
countries = gpd.read_file(src_dir / "ne_110m_admin_0_countries.shp")
merc, equal = countries.to_crs(3857).area / 1e6, countries.to_crs(8857).area / 1e6
for name in ("Greenland", "Brazil", "Antarctica"):
i = countries.index[countries.NAME == name][0]
print(f"{name:10} Web Mercator {merc[i]:>13,.0f} km² Equal Earth {equal[i]:>11,.0f} km²")Greenland Web Mercator 36,285,503 km² Equal Earth 2,206,879 km²
Brazil Web Mercator 9,059,541 km² Equal Earth 8,508,179 km²
Antarctica Web Mercator 9,204,514,350 km² Equal Earth 12,337,547 km²
That is the whole workflow. The rest of this article unpacks each stage so you can swap in your own processing safely instead of running it blind.
Separate folders, one function, errors per file
Keep inputs and outputs separate
Before any code, decide on two folders: one for source data you never touch, one for results. Writing processed files back into the source folder is how you lose the ability to re-run a batch, because your second run reads its own output as input.
from pathlib import Path
src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)mkdir(parents=True, exist_ok=True) creates the output folder (and any missing parents) and does nothing if it already exists, so it is safe to run every time.
Discover the files
Use pathlib.Path plus glob to find the files. Globbing one pattern at a time and chaining the results keeps things readable and lets you control exactly which extensions you accept. Sorting the result makes runs deterministic and the logs easy to read.
from pathlib import Path
src_dir = Path("data/raw")
PATTERNS = ("*.shp", "*.geojson", "*.gpkg")
files = sorted(p for pat in PATTERNS for p in src_dir.glob(pat))
print(f"Found {len(files)} files")
for f in files:
print(f.name)Found 17 files
lakes.shp
land.gpkg
ne_110m_admin_0_boundary_lines_land.shp
ne_110m_admin_0_countries.geojson
ne_110m_admin_0_countries.shp
ne_110m_admin_1_states_provinces.shp
ne_110m_coastline.shp
ne_110m_geography_regions_polys.shp
ne_110m_glaciated_areas.shp
ne_110m_lakes.shp
ne_110m_land.shp
ne_110m_ocean.shp
ne_110m_populated_places.shp
ne_110m_rivers_lake_centerlines.shp
ne_50m_admin_0_countries.shp
ne_50m_populated_places.shp
rivers.shp
Globbing *.shp deliberately ignores the .dbf, .shx, and .prj sidecars — GeoPandas reads them automatically from the .shp. If your data is nested in subfolders, swap glob for rglob; there is a dedicated guide on processing files in nested subfolders when that gets involved.
Read each file
Loop over the paths and load each into a GeoDataFrame with gpd.read_file(). The same call reads shapefiles, GeoJSON, GeoPackage, and every other format GDAL supports, which is what lets one loop handle a mixed folder.
import geopandas as gpd
for path in files:
gdf = gpd.read_file(path)
print(f"{path.name}: {len(gdf)} features, CRS {gdf.crs}")lakes.shp: 24 features, CRS None
land.gpkg: 127 features, CRS EPSG:4326
ne_110m_admin_0_boundary_lines_land.shp: 331 features, CRS EPSG:4326
ne_110m_admin_0_countries.geojson: 177 features, CRS EPSG:4326
ne_110m_admin_0_countries.shp: 177 features, CRS EPSG:4326
ne_110m_admin_1_states_provinces.shp: 51 features, CRS EPSG:4326
ne_110m_coastline.shp: 134 features, CRS EPSG:4326
ne_110m_geography_regions_polys.shp: 60 features, CRS EPSG:4326
ne_110m_glaciated_areas.shp: 11 features, CRS EPSG:4326
ne_110m_lakes.shp: 24 features, CRS EPSG:4326
ne_110m_land.shp: 127 features, CRS EPSG:4326
ne_110m_ocean.shp: 2 features, CRS EPSG:4326
ne_110m_populated_places.shp: 243 features, CRS EPSG:4326
ne_110m_rivers_lake_centerlines.shp: 13 features, CRS EPSG:4326
ne_50m_admin_0_countries.shp: 242 features, CRS EPSG:4326
ne_50m_populated_places.shp: 1251 features, CRS EPSG:4326
rivers.shp: 13 features, CRS EPSG:4326
At this point each gdf is an ordinary GeoDataFrame, so anything you would do to a single file works here too.
Apply one reusable processing function
This is the heart of the pattern. Put the repeated GIS logic in a single function that takes a GeoDataFrame and returns a GeoDataFrame. The loop stays fixed; you only ever edit this function.
def process_gdf(gdf):
gdf = gdf.copy()
if gdf.crs is None:
raise ValueError("Input has no CRS; cannot reproject safely")
gdf = gdf.to_crs("EPSG:4326") # standardise the CRS
gdf = gdf[gdf.geometry.notna()] # drop rows with no geometry
gdf["processed"] = True # tag the output
return gdfKeeping the transformation separate from the loop has a real payoff: you can test process_gdf() on one file until it is right, then run it across the whole folder unchanged. The same skeleton handles reprojecting, filtering, adding columns, clipping to a boundary, or standardising a schema — only the body changes.
Write to the output folder
Write each result to the output folder with a name you can trace back to the source. GeoPackage is the recommended output: one file per dataset, the CRS stored internally, no field-name limits.
written = []
try:
for path in files:
gdf = gpd.read_file(path)
result = process_gdf(gdf)
out_path = out_dir / f"{path.stem}.gpkg"
result.to_file(out_path, driver="GPKG")
written.append(out_path.name)
except ValueError as e:
print(f"stopped after {len(written)} of {len(files)} files, at {path.name}: {e}")stopped after 0 of 17 files, at lakes.shp: Input has no CRS; cannot reproject safely
(The try around the whole loop is only there to show where it stopped.) Sorted, the lakes delivered without a .prj come first, so the run ends before it has written anything.
Using path.stem keeps the original base name and just changes the extension, so roads.shp becomes roads.gpkg. See choosing an output format below for when a different format makes sense.
Handle errors per file and report
Wrap the read-process-write cycle for each file in its own try/except so a single bad file is logged and skipped rather than aborting the run. Collect one record per file and turn it into a report at the end.
import pandas as pd
summary = []
for path in files:
try:
gdf = gpd.read_file(path)
result = process_gdf(gdf)
result.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")
summary.append({"file": path.name, "rows": len(result), "status": "ok"})
except Exception as exc:
summary.append({"file": path.name, "rows": 0, "status": f"error: {exc}"})
report = pd.DataFrame(summary)
report.to_csv(out_dir / "_run_report.csv", index=False)
print(report.to_string(index=False)) file rows status
lakes.shp 0 error: Input has no CRS; cannot reproject safely
land.gpkg 127 ok
ne_110m_admin_0_boundary_lines_land.shp 331 ok
ne_110m_admin_0_countries.geojson 177 ok
ne_110m_admin_0_countries.shp 177 ok
ne_110m_admin_1_states_provinces.shp 51 ok
ne_110m_coastline.shp 134 ok
ne_110m_geography_regions_polys.shp 60 ok
ne_110m_glaciated_areas.shp 11 ok
ne_110m_lakes.shp 24 ok
ne_110m_land.shp 127 ok
ne_110m_ocean.shp 2 ok
ne_110m_populated_places.shp 243 ok
ne_110m_rivers_lake_centerlines.shp 13 ok
ne_50m_admin_0_countries.shp 242 ok
ne_50m_populated_places.shp 1251 ok
rivers.shp 7 ok
Note the rivers: 7 rows, ok. This process_gdf drops rows with no geometry, which is a reasonable rule for the odd empty record and exactly the wrong one for a file that lost half its geometries in transit — the report now shows a success with six rows fewer and no reason. Count what a rule removes, and fail past a threshold.
That summary is your audit trail. For richer, structured logging across a big job, see log and summarise errors in a batch job.
Processing a mixed folder of Natural Earth layers
The data used on this page
A real folder of shapefiles: thirteen layers of Natural Earth (public domain) at 1:110 million and 1:50 million, copied flat into data/raw/ with two other formats: the countries layer again as GeoJSON, and the land as a GeoPackage. Two more shapefiles are copies damaged on purpose in the two ways files most often arrive damaged — lakes.shp without its .prj, and rivers.shp cut off halfway through, as an interrupted copy leaves it. The target CRS is Equal Earth (EPSG:8857).
import io
import shutil
import zipfile
from pathlib import Path
import requests
NE = {"cultural": ["ne_110m_admin_0_countries", "ne_110m_admin_1_states_provinces", "ne_110m_populated_places",
"ne_110m_admin_0_boundary_lines_land", "ne_50m_admin_0_countries", "ne_50m_populated_places"],
"physical": ["ne_110m_coastline", "ne_110m_land", "ne_110m_ocean", "ne_110m_rivers_lake_centerlines",
"ne_110m_lakes", "ne_110m_glaciated_areas", "ne_110m_geography_regions_polys"]}
SRC = Path("data/ne")
shutil.rmtree("data/processed", ignore_errors=True) # every run of the page starts from nothing
for theme, names in NE.items():
for name in names:
if not (SRC / theme / f"{name}.shp").exists():
scale = name.split("_")[1]
reply = requests.get(f"https://naciscdn.org/naturalearth/{scale}/{theme}/{name}.zip", timeout=120)
reply.raise_for_status()
zipfile.ZipFile(io.BytesIO(reply.content)).extractall(SRC / theme)
# two deliveries damaged on purpose: a missing .prj, and a .shp cut off mid-copy
bad = SRC / "delivered"
bad.mkdir(exist_ok=True)
for ext in ("shp", "shx", "dbf"):
shutil.copy(SRC / "physical" / f"ne_110m_lakes.{ext}", bad / f"lakes.{ext}")
for ext in ("shp", "shx", "dbf", "prj"):
shutil.copy(SRC / "physical" / f"ne_110m_rivers_lake_centerlines.{ext}", bad / f"rivers.{ext}")
data = (bad / "rivers.shp").read_bytes()
(bad / "rivers.shp").write_bytes(data[: len(data) // 2])
# one flat folder of mixed formats, as a supplier would send it
import geopandas as gpd
raw = Path("data/raw")
shutil.rmtree(raw, ignore_errors=True)
raw.mkdir(parents=True)
for f in SRC.rglob("*"):
if f.is_file() and f.suffix in (".shp", ".shx", ".dbf", ".prj", ".cpg"):
shutil.copy(f, raw / f.name)
gpd.read_file(SRC / "cultural" / "ne_110m_admin_0_countries.shp").to_file(raw / "ne_110m_admin_0_countries.geojson")
gpd.read_file(SRC / "physical" / "ne_110m_land.shp").to_file(raw / "land.gpkg")
print(sorted(p.suffix for p in raw.iterdir() if p.suffix in (".shp", ".geojson", ".gpkg")).count(".shp"), "shapefiles,",
"and", [p.name for p in sorted(raw.glob("*.geojson")) + sorted(raw.glob("*.gpkg"))])15 shapefiles, and ['ne_110m_admin_0_countries.geojson', 'land.gpkg']
Example 1: The minimal discover-and-loop
The smallest thing that works: find every shapefile, read it, write a copy to the output folder. Start here and add logic to the middle.
from pathlib import Path
import geopandas as gpd
src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)
for path in sorted(src_dir.glob("*.shp")):
gdf = gpd.read_file(path)
gdf.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")
print(f"Wrote {path.stem}.gpkg")Wrote lakes.gpkg
Wrote ne_110m_admin_0_boundary_lines_land.gpkg
Wrote ne_110m_admin_0_countries.gpkg
Wrote ne_110m_admin_1_states_provinces.gpkg
Wrote ne_110m_coastline.gpkg
Wrote ne_110m_geography_regions_polys.gpkg
Wrote ne_110m_glaciated_areas.gpkg
Wrote ne_110m_lakes.gpkg
Wrote ne_110m_land.gpkg
Wrote ne_110m_ocean.gpkg
Wrote ne_110m_populated_places.gpkg
Wrote ne_110m_rivers_lake_centerlines.gpkg
Wrote ne_50m_admin_0_countries.gpkg
Wrote ne_50m_populated_places.gpkg
Wrote rivers.gpkg
Example 2: A reusable process_gdf() you can swap out
Isolating the transformation is the single most useful habit in batch GIS. The loop below never changes no matter what process_gdf() does.
from pathlib import Path
import geopandas as gpd
src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)
def process_gdf(gdf):
"""Reproject, keep only valid polygons, add an area column."""
gdf = gdf.copy()
if gdf.crs is None:
raise ValueError("Missing CRS")
gdf = gdf.to_crs("EPSG:8857") # Equal Earth: metres, and areas that are right
gdf = gdf[gdf.geometry.is_valid]
gdf["area_m2"] = gdf.geometry.area
return gdf
# the Natural Earth originals only; the damaged deliveries need Example 4's handling
for path in sorted(src_dir.glob("ne_*.shp")):
result = process_gdf(gpd.read_file(path))
result.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")Example 3: Support multiple input formats at once
Real folders are rarely one format. Chain several glob patterns so a single run picks up shapefiles, GeoJSON, and GeoPackages together.
from pathlib import Path
import geopandas as gpd
src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)
PATTERNS = ("*.shp", "*.geojson", "*.json", "*.gpkg")
files = sorted(p for pat in PATTERNS for p in src_dir.glob(pat))
print(f"Found {len(files)} files across {len(PATTERNS)} patterns")
for path in files:
gdf = gpd.read_file(path)
if gdf.crs is None:
print(f"skipped {path.name}: no CRS")
continue
gdf = gdf.to_crs("EPSG:4326")
gdf.to_file(out_dir / f"{path.stem}_{path.suffix[1:]}.gpkg", driver="GPKG")Found 17 files across 4 patterns
skipped lakes.shp: no CRS
Because gpd.read_file() detects the format from the file itself, the body of the loop does not care whether it is reading a shapefile or a GeoJSON.
Example 4: try/except with a collected summary list
The production shape: every file is wrapped, failures are recorded with the exception message, and the loop always finishes.
from pathlib import Path
import geopandas as gpd
src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)
def process_gdf(gdf):
if gdf.crs is None:
raise ValueError("Missing CRS")
return gdf.to_crs("EPSG:4326")
summary = []
for path in sorted(src_dir.glob("*.shp")):
print(f"Processing {path.name} ...")
try:
result = process_gdf(gpd.read_file(path))
result.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")
summary.append((path.name, len(result), "ok"))
except Exception as exc:
summary.append((path.name, 0, f"error: {exc}"))
ok = sum(1 for _, _, s in summary if s == "ok")
print(f"\nDone: {ok}/{len(summary)} succeeded")Processing lakes.shp ...
Processing ne_110m_admin_0_boundary_lines_land.shp ...
Processing ne_110m_admin_0_countries.shp ...
Processing ne_110m_admin_1_states_provinces.shp ...
Processing ne_110m_coastline.shp ...
Processing ne_110m_geography_regions_polys.shp ...
Processing ne_110m_glaciated_areas.shp ...
Processing ne_110m_lakes.shp ...
Processing ne_110m_land.shp ...
Processing ne_110m_ocean.shp ...
Processing ne_110m_populated_places.shp ...
Processing ne_110m_rivers_lake_centerlines.shp ...
Processing ne_50m_admin_0_countries.shp ...
Processing ne_50m_populated_places.shp ...
Processing rivers.shp ...
Done: 14/15 succeeded
Example 5: Write a CSV run-report with pandas
Turn the collected records into a DataFrame and save a CSV alongside the outputs. This is the artefact you keep — it says what ran, when, and how it went.
from pathlib import Path
from datetime import datetime
import geopandas as gpd
import pandas as pd
src_dir = Path("data/raw")
out_dir = Path("data/processed")
out_dir.mkdir(parents=True, exist_ok=True)
rows = []
for path in sorted(src_dir.glob("*.shp")):
try:
gdf = gpd.read_file(path)
out = gdf.to_crs("EPSG:4326")
out.to_file(out_dir / f"{path.stem}.gpkg", driver="GPKG")
rows.append({"file": path.name, "features": len(out),
"crs_in": str(gdf.crs), "status": "ok"})
except Exception as exc:
rows.append({"file": path.name, "features": 0,
"crs_in": "-", "status": f"error: {exc}"})
report = pd.DataFrame(rows)
report["run_at"] = datetime.now().isoformat(timespec="seconds")
report.to_csv(out_dir / "_run_report.csv", index=False)
print(report.to_string(index=False)) file features crs_in status run_at
lakes.shp 0 - error: Cannot transform naive geometries. Please set a crs on the object first. 2026-10-10T15:00:49
ne_110m_admin_0_boundary_lines_land.shp 331 EPSG:4326 ok 2026-10-10T15:00:49
ne_110m_admin_0_countries.shp 177 EPSG:4326 ok 2026-10-10T15:00:49
ne_110m_admin_1_states_provinces.shp 51 EPSG:4326 ok 2026-10-10T15:00:49
ne_110m_coastline.shp 134 EPSG:4326 ok 2026-10-10T15:00:49
ne_110m_geography_regions_polys.shp 60 EPSG:4326 ok 2026-10-10T15:00:49
ne_110m_glaciated_areas.shp 11 EPSG:4326 ok 2026-10-10T15:00:49
ne_110m_lakes.shp 24 EPSG:4326 ok 2026-10-10T15:00:49
ne_110m_land.shp 127 EPSG:4326 ok 2026-10-10T15:00:49
ne_110m_ocean.shp 2 EPSG:4326 ok 2026-10-10T15:00:49
ne_110m_populated_places.shp 243 EPSG:4326 ok 2026-10-10T15:00:49
ne_110m_rivers_lake_centerlines.shp 13 EPSG:4326 ok 2026-10-10T15:00:49
ne_50m_admin_0_countries.shp 242 EPSG:4326 ok 2026-10-10T15:00:49
ne_50m_populated_places.shp 1251 EPSG:4326 ok 2026-10-10T15:00:49
rivers.shp 13 EPSG:4326 ok 2026-10-10T15:00:49
The four parts of every folder job
This pattern works because nearly every folder-based GIS job decomposes into the same four moving parts: a way to find the files, a loop over them, a function that transforms one GeoDataFrame, and an output folder for the results. Once you see a batch job that way, the loop becomes boilerplate you can reuse forever, and all your thinking goes into the one function in the middle.
Separating the transformation from the loop is what makes the whole thing testable. You develop process_gdf() against a single representative file, in a notebook or the REPL, until it does exactly what you want. Only then do you point the loop at the whole folder. If something breaks later, you already know the loop is sound, so the bug is in the function — a much smaller place to look.
The per-file try/except matters more than it looks. Real folders always contain at least one surprise: an empty file, a broken geometry, a shapefile that lost its .prj. Without the guard, that one file throws an exception that unwinds the entire loop and you lose every result computed before it. With the guard, the bad file becomes a single row in your report and the batch runs to completion. The summary list then does double duty — it is both your progress log and the record you keep for later.
GeoPandas is the right tool when you have a moderate number of vector files and standard operations. When the folder gets large enough that runtime hurts, reach for parallel processing; when a job is long enough that you need it to survive an interruption, make it resumable. Those are refinements of this same skeleton, not replacements for it.
CRS, geometry types and formats across a folder
Files with a missing or wrong CRS
If gdf.crs is None, calling to_crs() raises an error because there is no source projection to transform from. You must first attach the correct CRS with set_crs() — and you need to actually know what it is. Standardising a mixed folder before processing is a job in its own right; work through the Python GIS data cleaning checklist to catch missing CRS, invalid geometries, and schema drift up front.
Mixed geometry types in one folder
A folder can hold points, lines, and polygons. Some operations only make sense for one type — polygon area is meaningless for points, and writing mixed types into a single layer can fail. If your process_gdf() assumes polygons, guard it with a check on gdf.geom_type and skip or branch on anything unexpected.
Choosing the output format
Prefer GeoPackage (driver="GPKG") for output. Shapefiles split one dataset across four-plus files, truncate field names to 10 characters, cap at 2 GB, and store the CRS in a fragile .prj sidecar. GeoPackage is a single SQLite file with no field-name limits, an internal CRS, and support for many layers. Use GeoJSON only when a downstream tool specifically needs it. If you need several formats from one run, see batch export to multiple formats.
One output file per input, or one combined file
The examples here write one output per input, which keeps a clear trace back to the source. If your goal is instead to stitch everything into a single dataset, that is a merge, not a per-file write — collect the frames and concatenate them. See merge many shapefiles into one file for the correct CRS-safe way to do it.
Large folders and memory
gpd.read_file() loads a whole file into memory. Processing one file at a time inside the loop (as shown) keeps only one GeoDataFrame resident, so even a big folder stays manageable as long as no single file is enormous. If an individual file is too large to fit, read it in chunks with pyogrio options rather than loading it whole.
Rasters are a different pipeline
This workflow is for vector data read through GeoPandas. Rasters need Rasterio and a windowed read/write pattern, not read_file() and to_file(). If your folder is full of GeoTIFFs, go straight to batch process rasters with Rasterio.
Internal links
- How to Batch Process Multiple Shapefiles in Python
- Batch Convert Shapefiles to GeoPackage
- Batch Clip Many Layers to a Boundary
- Merge Many Shapefiles into One File
- Batch Export to Multiple Formats
- Speed Up Batch Jobs with Parallel Processing
- Build a Resumable Batch Job
- Log and Summarise Errors in a Batch Job
- Batch Process Rasters with Rasterio
- Process Files in Nested Subfolders
- The Python GIS Data Cleaning Checklist
FAQ
How do I loop through every GIS file in a folder in Python?
Use pathlib with glob. For a single format, Path("data/raw").glob("*.shp") yields every shapefile. For several formats, chain patterns: sorted(p for pat in ("*.shp", "*.geojson", "*.gpkg") for p in src_dir.glob(pat)). Sorting makes the run order deterministic, which keeps logs readable and results reproducible.
Should I put my processing logic in a function?
Yes. Keeping the transformation in a single process_gdf(gdf) function separates the what (your GIS logic) from the how (the file loop). You can test the function on one file until it is correct, then run it across the folder without changing the loop. It also makes the code far easier to read and to reuse in the next batch job.
Can I mix shapefiles, GeoJSON, and GeoPackage in one run?
Yes. Chain multiple glob patterns to discover them and let gpd.read_file() detect each format automatically. The body of your loop does not need to know or care which format it is reading, because every file becomes an ordinary GeoDataFrame after the read.
When should I use the specialised batch guides instead of this one?
This article is the general pattern. Move to a specialised guide when your job has a specific shape: parallel processing for speed on big folders, a resumable job for long runs that must survive interruption, nested subfolders for tree-structured data, merging to combine into one file, or Rasterio for raster data. Each is a variation on the loop taught here.
How do I keep a record of what the batch did?
Collect one dictionary per file inside the loop — file name, feature count, status — then build a pandas.DataFrame and write it to CSV alongside your outputs. That report tells you which files succeeded, which failed and why, and how many features each produced. For structured, timestamped logging on larger jobs, see log and summarise errors in a batch job.