How to Automate GIS Workflows with Python

On this page

Problem statement

Many GIS tasks are repetitive: open a file, check the CRS, reproject it, clean field names, remove bad records, and save a new copy. That works for one dataset, but it becomes slow and error-prone when you need to process 10, 50, or 200 files.

Files from different sources need standardising to one CRS, trimming to the fields that matter, clearing of empty geometries and writing to a separate folder — the same four steps, every time, for every file. Written once as a GeoPandas function, they run identically on the tenth file and the two-hundredth.

Quick answer

To automate a GIS workflow in Python:

define the repeated steps
collect input files from a folder
process each file with the same GeoPandas operations
save outputs to a separate folder

Typical tools:

Python
GeoPandas
pathlib

A basic automation script can batch process shapefiles and GeoJSON files, reproject them, clean attributes, filter bad geometry records, and write standardized outputs automatically.

Batch processing pipeline

The per-file batch processing pipeline.

From a list of steps to a reusable function

The last two are worth honouring — automating an unstable process just moves the work.

Define the workflow first

Before writing code, list the exact steps you want to repeat. A simple vector workflow might be:

load shapefiles or GeoJSON files
check whether each file has a CRS
reproject to a target CRS such as EPSG:3857
standardize column names
drop rows with missing required values
remove null, empty, or invalid geometries
save processed files to a new folder

This matters because automation works best when the workflow is explicit and consistent.

Set up the Python environment

Install GeoPandas if needed:

pip install geopandas

Imports used in this tutorial:

from pathlib import Path
import geopandas as gpd

If you use Python regularly for GIS, keep a dedicated environment for geospatial packages.

Read vector files from an input folder

Use pathlib to collect files instead of hard-coding names.

from pathlib import Path

input_folder = Path("data/raw")
output_folder = Path("data/processed")
output_folder.mkdir(parents=True, exist_ok=True)

input_files = list(input_folder.glob("*.shp")) + list(input_folder.glob("*.geojson"))

print(f"Found {len(input_files)} files")
for file_path in input_files:
    print(file_path.name)

Found 16 files
ne_110m_ocean.shp
ne_50m_populated_places.shp
ne_110m_rivers_lake_centerlines.shp
ne_110m_admin_1_states_provinces.shp
ne_110m_admin_0_countries.shp
ne_110m_admin_0_boundary_lines_land.shp
rivers.shp
ne_110m_populated_places.shp
ne_110m_coastline.shp
ne_110m_lakes.shp
ne_110m_glaciated_areas.shp
lakes.shp
ne_110m_land.shp
ne_50m_admin_0_countries.shp
ne_110m_geography_regions_polys.shp
ne_110m_admin_0_countries.geojson

This pattern is useful for batch GIS work because you can drop new files into one folder and rerun the same script.

Apply the same processing steps to each file

The script below does four practical things:

skips files with missing CRS metadata
reprojects valid files to a target CRS
standardizes column names
removes bad geometry records and exports cleaned output

from pathlib import Path
import geopandas as gpd

input_folder = Path("data/raw")
output_folder = Path("data/processed")
output_folder.mkdir(parents=True, exist_ok=True)

target_crs = "EPSG:3857"

input_files = list(input_folder.glob("*.shp")) + list(input_folder.glob("*.geojson"))

for file_path in input_files:
    gdf = gpd.read_file(file_path)
    n_read = len(gdf)

    if gdf.crs is None:
        print(f"Skipping {file_path.name}: missing CRS")
        continue

    if gdf.crs.to_string() != target_crs:
        gdf = gdf.to_crs(target_crs)

    # standardize column names
    gdf.columns = [col.lower().replace(" ", "_") for col in gdf.columns]

    # remove empty or null geometries
    gdf = gdf[gdf.geometry.notnull()]
    gdf = gdf[~gdf.geometry.is_empty]

    # drop invalid geometries
    gdf = gdf[gdf.is_valid]

    # drop rows with null values in a required field if it exists
    if "name" in gdf.columns:
        gdf = gdf[gdf["name"].notna()]

    output_path = output_folder / f"{file_path.stem}_processed.geojson"
    gdf.to_file(output_path, driver="GeoJSON")

    dropped = n_read - len(gdf)
    print(f"Processed: {file_path.name} -> {output_path.name}" + (f"  ({dropped} of {n_read} rows dropped)" if dropped else ""))

Processed: ne_110m_ocean.shp -> ne_110m_ocean_processed.geojson
Processed: ne_50m_populated_places.shp -> ne_50m_populated_places_processed.geojson
Processed: ne_110m_rivers_lake_centerlines.shp -> ne_110m_rivers_lake_centerlines_processed.geojson
Processed: ne_110m_admin_1_states_provinces.shp -> ne_110m_admin_1_states_provinces_processed.geojson
Processed: ne_110m_admin_0_countries.shp -> ne_110m_admin_0_countries_processed.geojson  (1 of 177 rows dropped)
Processed: ne_110m_admin_0_boundary_lines_land.shp -> ne_110m_admin_0_boundary_lines_land_processed.geojson  (331 of 331 rows dropped)
Processed: rivers.shp -> rivers_processed.geojson  (6 of 13 rows dropped)
Processed: ne_110m_populated_places.shp -> ne_110m_populated_places_processed.geojson
Processed: ne_110m_coastline.shp -> ne_110m_coastline_processed.geojson
Processed: ne_110m_lakes.shp -> ne_110m_lakes_processed.geojson
Processed: ne_110m_glaciated_areas.shp -> ne_110m_glaciated_areas_processed.geojson
Skipping lakes.shp: missing CRS
Processed: ne_110m_land.shp -> ne_110m_land_processed.geojson  (1 of 127 rows dropped)
Processed: ne_50m_admin_0_countries.shp -> ne_50m_admin_0_countries_processed.geojson
Processed: ne_110m_geography_regions_polys.shp -> ne_110m_geography_regions_polys_processed.geojson
Processed: ne_110m_admin_0_countries.geojson -> ne_110m_admin_0_countries_processed.geojson  (1 of 177 rows dropped)

The drop counts are the part to read, and the version of this loop without them prints sixteen identical success lines. Every one of the rules removed something real:

the required-field rule emptied a whole layer. Natural Earth's land boundary lines have a NAME column that is blank on every row, so name must not be null dropped all 331 and wrote an empty file.
the validity rule dropped Antarctica. Projected to Web Mercator, which cannot show the poles, it is no longer a valid polygon — one row from the countries and one from the land, gone without a trace.
the null-geometry rule hid a damaged file. The rivers file cut off mid-copy lost 6 of its 13 rows and was reported as processed.
two inputs wrote one output. The countries shapefile and its GeoJSON copy have the same stem, so the second silently replaced the first.

Only the missing .prj produced a message. A cleaning rule that drops rows should say how many, per file, every run.

This gives you a repeatable pattern for routine vector data processing.

Save outputs safely

A few details in the script are important:

output_folder.mkdir(parents=True, exist_ok=True) creates the output folder if needed
outputs use a new suffix such as _processed
original files stay unchanged
exporting to GeoJSON avoids some shapefile limitations during automation

If you need shapefile output instead:

output_path = output_folder / f"{file_path.stem}_processed.shp"
gdf.to_file(output_path)

Wrap the workflow in a reusable function

If you plan to run the workflow more than once, turn it into a function.

from pathlib import Path
import geopandas as gpd

def process_vector_files(input_folder, output_folder, target_crs="EPSG:3857"):
    input_folder = Path(input_folder)
    output_folder = Path(output_folder)
    output_folder.mkdir(parents=True, exist_ok=True)

    input_files = list(input_folder.glob("*.shp")) + list(input_folder.glob("*.geojson"))

    for file_path in input_files:
        gdf = gpd.read_file(file_path)

        if gdf.crs is None:
            print(f"Skipping {file_path.name}: missing CRS")
            continue

        if gdf.crs.to_string() != target_crs:
            gdf = gdf.to_crs(target_crs)

        gdf.columns = [col.lower().replace(" ", "_") for col in gdf.columns]
        gdf = gdf[gdf.geometry.notnull()]
        gdf = gdf[~gdf.geometry.is_empty]
        gdf = gdf[gdf.is_valid]

        if "name" in gdf.columns:
            gdf = gdf[gdf["name"].notna()]

        output_path = output_folder / f"{file_path.stem}_processed.geojson"
        gdf.to_file(output_path, driver="GeoJSON")

        print(f"Saved {output_path}")

if __name__ == "__main__":
    process_vector_files("data/raw", "data/processed", target_crs="EPSG:3857")

process_vector_files("data/raw", "data/processed", target_crs="EPSG:3857")    # what __main__ would run

Saved data/processed/ne_110m_ocean_processed.geojson
Saved data/processed/ne_50m_populated_places_processed.geojson
Saved data/processed/ne_110m_rivers_lake_centerlines_processed.geojson
Saved data/processed/ne_110m_admin_1_states_provinces_processed.geojson
Saved data/processed/ne_110m_admin_0_countries_processed.geojson
Saved data/processed/ne_110m_admin_0_boundary_lines_land_processed.geojson
Saved data/processed/rivers_processed.geojson
Saved data/processed/ne_110m_populated_places_processed.geojson
Saved data/processed/ne_110m_coastline_processed.geojson
Saved data/processed/ne_110m_lakes_processed.geojson
Saved data/processed/ne_110m_glaciated_areas_processed.geojson
Skipping lakes.shp: missing CRS
Saved data/processed/ne_110m_land_processed.geojson
Saved data/processed/ne_50m_admin_0_countries_processed.geojson
Saved data/processed/ne_110m_geography_regions_polys_processed.geojson
Saved data/processed/ne_110m_admin_0_countries_processed.geojson

This structure is easier to reuse in scheduled jobs, command-line scripts, or larger data pipelines.

Automating a folder of Natural Earth layers

The data used on this page

A real folder of shapefiles: thirteen layers of Natural Earth (public domain) at 1:110 million and 1:50 million, copied flat into data/raw/ with two other formats: the countries layer again as GeoJSON, and the land as a GeoPackage. Two more shapefiles are copies damaged on purpose in the two ways files most often arrive damaged — lakes.shp without its .prj, and rivers.shp cut off halfway through, as an interrupted copy leaves it. The target CRS is Equal Earth (EPSG:8857).

import io
import shutil
import zipfile
from pathlib import Path

import requests

NE = {"cultural": ["ne_110m_admin_0_countries", "ne_110m_admin_1_states_provinces", "ne_110m_populated_places",
                   "ne_110m_admin_0_boundary_lines_land", "ne_50m_admin_0_countries", "ne_50m_populated_places"],
      "physical": ["ne_110m_coastline", "ne_110m_land", "ne_110m_ocean", "ne_110m_rivers_lake_centerlines",
                   "ne_110m_lakes", "ne_110m_glaciated_areas", "ne_110m_geography_regions_polys"]}
SRC = Path("data/ne")
for d in ("data/processed", "data/exported", "data/reprojected"):
    shutil.rmtree(d, ignore_errors=True)               # every run of the page starts from nothing
for theme, names in NE.items():
    for name in names:
        if not (SRC / theme / f"{name}.shp").exists():
            scale = name.split("_")[1]
            reply = requests.get(f"https://naciscdn.org/naturalearth/{scale}/{theme}/{name}.zip", timeout=120)
            reply.raise_for_status()
            zipfile.ZipFile(io.BytesIO(reply.content)).extractall(SRC / theme)

# two deliveries damaged on purpose: a missing .prj, and a .shp cut off mid-copy
bad = SRC / "delivered"
bad.mkdir(exist_ok=True)
for ext in ("shp", "shx", "dbf"):
    shutil.copy(SRC / "physical" / f"ne_110m_lakes.{ext}", bad / f"lakes.{ext}")
for ext in ("shp", "shx", "dbf", "prj"):
    shutil.copy(SRC / "physical" / f"ne_110m_rivers_lake_centerlines.{ext}", bad / f"rivers.{ext}")
data = (bad / "rivers.shp").read_bytes()
(bad / "rivers.shp").write_bytes(data[: len(data) // 2])

# one flat folder of mixed formats, as a supplier would send it
import geopandas as gpd
raw = Path("data/raw")
shutil.rmtree(raw, ignore_errors=True)
raw.mkdir(parents=True)
for f in SRC.rglob("*"):
    if f.is_file() and f.suffix in (".shp", ".shx", ".dbf", ".prj", ".cpg"):
        shutil.copy(f, raw / f.name)
gpd.read_file(SRC / "cultural" / "ne_110m_admin_0_countries.shp").to_file(raw / "ne_110m_admin_0_countries.geojson")
gpd.read_file(SRC / "physical" / "ne_110m_land.shp").to_file(raw / "land.gpkg")
print(sorted(p.suffix for p in raw.iterdir() if p.suffix in (".shp", ".geojson", ".gpkg")).count(".shp"), "shapefiles,",
      "and", [p.name for p in sorted(raw.glob("*.geojson")) + sorted(raw.glob("*.gpkg"))])

15 shapefiles, and ['ne_110m_admin_0_countries.geojson', 'land.gpkg']

Example 1: Batch read and export vector files

from pathlib import Path
import geopandas as gpd

input_folder = Path("data/raw")
output_folder = Path("data/exported")
output_folder.mkdir(parents=True, exist_ok=True)

for file_path in list(input_folder.glob("*.shp")) + list(input_folder.glob("*.geojson")):
    gdf = gpd.read_file(file_path)
    output_path = output_folder / f"{file_path.stem}.geojson"
    gdf.to_file(output_path, driver="GeoJSON")

print(len(list(Path('data/exported').glob('*.geojson'))), 'GeoJSON files written')

15 GeoJSON files written

Example 2: Reproject all files to one CRS

from pathlib import Path
import geopandas as gpd

target_crs = "EPSG:8857"                 # Equal Earth: these layers are global
output_folder = Path("data/reprojected")
output_folder.mkdir(parents=True, exist_ok=True)

for file_path in Path("data/raw").glob("*.shp"):
    gdf = gpd.read_file(file_path)
    if gdf.crs is None:
        continue
    if gdf.crs.to_string() != target_crs:
        gdf = gdf.to_crs(target_crs)
    gdf.to_file(output_folder / f"{file_path.stem}_equal_earth.geojson", driver="GeoJSON")
print(len(list(output_folder.glob("*.geojson"))), "files reprojected")

14 files reprojected

Example 3: Clean attributes during automation

import geopandas as gpd

gdf = gpd.read_file("data/raw/ne_110m_populated_places.shp")
gdf = gdf.rename(columns={"NAME": "name", "POP_MAX": "pop_max"})
gdf = gdf[["name", "pop_max", "geometry"]]
gdf = gdf[gdf["name"].notna()]
gdf.to_file("data/processed/places_clean.geojson", driver="GeoJSON")
print(gdf.shape, list(gdf.columns))

(243, 3) ['name', 'pop_max', 'geometry']

Example 4: Wrap a cleaning step in a reusable function

from pathlib import Path
import geopandas as gpd

def clean_layer(file_path, output_folder, target_crs):
    gdf = gpd.read_file(file_path)
    if gdf.crs is None:
        return

    if gdf.crs.to_string() != target_crs:
        gdf = gdf.to_crs(target_crs)

    gdf.columns = [c.lower().replace(" ", "_") for c in gdf.columns]
    gdf = gdf[gdf.geometry.notnull() & ~gdf.geometry.is_empty & gdf.is_valid]

    output_folder = Path(output_folder)
    output_folder.mkdir(parents=True, exist_ok=True)

    output_path = output_folder / f"{Path(file_path).stem}_clean.geojson"
    gdf.to_file(output_path, driver="GeoJSON")

clean_layer("data/raw/ne_110m_land.shp", "data/processed", "EPSG:3857")
print(len(gpd.read_file("data/raw/ne_110m_land.shp")), "land polygons in,",
      len(gpd.read_file("data/processed/ne_110m_land_clean.geojson")), "out")

127 land polygons in, 126 out

Repeating rules instead of clicks

GIS automation with Python is useful when you need to apply the same rules to many datasets. Instead of repeating the same clicks in desktop software, you write the workflow once and run it on every file.

Most GIS work is well served by level two; level four is for jobs nobody watches.

GeoPandas is a good fit for this type of vector workflow because it handles:

reading and writing common vector formats
column and attribute cleanup
CRS conversion with to_crs()
filtering rows based on geometry or attribute rules

This approach works well for:

repeated vector data cleaning
standardizing data from multiple sources
routine reporting workflows
small to medium batch jobs

It is also easier to audit. When the logic is in code, you can see exactly how each dataset was processed.

Missing CRSs, invalid geometry and large inputs

Missing CRS metadata

If gdf.crs is None, do not guess unless you know the source CRS. A wrong CRS assignment can break the rest of the workflow. Skip those files and fix the metadata first.

Different CRS values across input files

Different source CRS values are normal. The important part is that each file has correct CRS metadata before reprojection.

Invalid geometries

The line below removes invalid features:

gdf = gdf[gdf.is_valid]

That is useful for cleaning, but it drops features rather than repairing them. If you need to keep invalid features, use a geometry repair step instead of filtering them out.

Area calculations need the right projected CRS

EPSG:3857 is convenient for web mapping, but it is not a good default for accurate area calculations. If you need area values, reproject to an appropriate projected CRS for your region and analysis before using gdf.area.

Shapefile limitations

Shapefiles have field name length limits and more restrictive data typing. For automated workflows, GeoJSON or GeoPackage is often easier to work with as an output format.

Large datasets

GeoPandas works well for many file-based workflows, but large datasets can use a lot of memory. For bigger jobs, consider formats and tools such as GeoPackage, Parquet, or PostGIS.

Avoid overwriting source data

Write outputs to a separate folder and use clear file name suffixes such as _processed or _clean. That reduces the risk of accidental data loss.

Internal links

For a broader overview, see Python GIS automation basics.

Related task guides:

If you need to export cleaned outputs, see How to Export GeoJSON in Python with GeoPandas.

FAQ

How do I automate multiple shapefiles in Python?

Use pathlib to collect all .shp files in a folder, loop through them with GeoPandas, apply the same processing steps, and save each result to an output folder.

What Python library is best for GIS workflow automation?

For vector file workflows, GeoPandas is usually the best starting point. It covers reading, writing, reprojection, filtering, and attribute cleanup in one workflow.

Can I automate reprojection and attribute cleaning in one script?

Yes. A single GeoPandas script can read a file, reproject it with to_crs(), rename fields, drop rows, filter invalid geometries, and export the cleaned result.

How do I log errors so one bad file does not stop the whole batch?

Wrap each file's processing in a try/except inside the loop and print or log the failure, then continue. That way a single corrupt or unreadable file is recorded and skipped instead of crashing the entire run.

Can I process files in subfolders recursively?

Yes. Use Path("data/raw").rglob("*.shp") instead of glob(). rglob walks every subdirectory, which is useful when input files are organized into per-region or per-date folders.

How do I write all processed layers into a single GeoPackage instead of separate files?

Call to_file() on the same .gpkg path and pass a distinct layer= name for each file:

gdf.to_file("data/processed/output.gpkg", layer=file_path.stem, driver="GPKG")

A GeoPackage can hold many layers, which keeps batch outputs in one file.