PostGIS Explained: When a Spatial Database Beats a Folder of Files
On this page
Problem statement
The folder started with three shapefiles. It now has ninety, plus parcels_v2_FINAL.gpkg, plus a backup directory nobody dares delete. Two people edit it. The nightly job reads a file that a colleague is halfway through rewriting.
data/
βββ parcels_2024.shp # which one is current?
βββ parcels_2024_fixed.shp
βββ parcels_v2_FINAL.gpkg
βββ parcels_v2_FINAL_v3.gpkg
βββ backup/
Files are an excellent way to store spatial data and a poor way to manage it. A folder of them has no concurrency control, no schema enforcement, and no way to answer "which parcels changed last week" without opening all of them.
PostGIS is the usual next step. But it is not a straight upgrade β it adds a server to run, a schema to design, and a connection to configure. Knowing when it pays for itself is more useful than knowing how to install it.
The measurements on this page use HM Land Registry's 338,611 freehold parcels for Birmingham, the city's 69 wards, and its OpenStreetMap buildings and schools, held once as GeoPackages and once in PostGIS 3.4.
Quick answer
Move to PostGIS when at least two of these are true:
- More than one person or process writes the data. Files have no locking worth the name; a database has transactions.
- The data does not fit comfortably in memory. SQL filters before Python loads; a plain
read_file()does not. - You need types and rules that files cannot hold. A shapefile has no date-time type and a 10-character column-name limit; no file format rejects a bad row at write time.
- The interesting questions are relational. Joining parcels to wards to buildings to schools is one SQL query and four file reads.
- You need history. A table with a valid-from column beats forty files named after dates.
Stay on files when the data is read-only, fits in memory, has one writer, and is handed to people who open it in QGIS.
The shape of the difference is where the filter runs. Here are the parcels of more than a hectare in Birmingham's Ladywood ward, three ways:
import time
import geopandas as gpd
ward = gpd.read_file("data/wards.gpkg", where="WD24NM = 'Ladywood'").geometry.iloc[0]
# files: read everything, then filter
start = time.perf_counter()
parcels = gpd.read_file("data/parcels.gpkg")
big = parcels[parcels.within(ward) & (parcels.area > 10_000)]
print(f"read all: {len(parcels):>7,} rows read, {len(big)} kept, {time.perf_counter() - start:.2f} s")
# files, but let the GeoPackage's R-tree do the first cut
start = time.perf_counter()
nearby = gpd.read_file("data/parcels.gpkg", mask=ward)
big = nearby[nearby.within(ward) & (nearby.area > 10_000)]
print(f"GeoPackage mask: {len(nearby):>5,} rows read, {len(big)} kept, {time.perf_counter() - start:.2f} s")
# PostGIS: filter, then read
start = time.perf_counter()
big = gpd.read_postgis(
"""
SELECT p."INSPIREID", p.geom
FROM bham_parcels p JOIN bham_wards w ON ST_Within(p.geom, w.geom)
WHERE w."WD24NM" = %(ward)s AND ST_Area(p.geom) > 10000
""",
engine, geom_col="geom", params={"ward": "Ladywood"},
)
print(f"PostGIS: {len(big):>5,} rows sent, {len(big)} kept, {time.perf_counter() - start:.2f} s")
read all: 338,611 rows read, 51 kept, 0.47 s
GeoPackage mask: 2,622 rows read, 51 kept, 0.02 s
PostGIS: 51 rows sent, 51 kept, 0.02 s
Reading everything and filtering in pandas is the slow one. But notice the middle line: a GeoPackage carries an R-tree too, and a mask pushes the spatial filter into it. For one spatial filter on one table, a well-used GeoPackage is about as quick as the database. The reasons to move are the other four on the list.
What each one is actually good at
What PostGIS is, the index, where GeoPandas fits
What PostGIS actually is
PostgreSQL is a relational database. PostGIS is an extension that teaches it about geometry: a geometry column type, several hundred ST_ functions, and spatial indexes.
print(sql("DROP TABLE IF EXISTS typed_parcels"))
print(sql("""
CREATE TABLE typed_parcels (
parcel_id bigserial PRIMARY KEY,
inspire_id bigint NOT NULL UNIQUE,
surveyed date,
area_m2 double precision GENERATED ALWAYS AS (ST_Area(geom)) STORED,
geom geometry(Polygon, 27700) NOT NULL
);
CREATE INDEX typed_parcels_geom_idx ON typed_parcels USING GIST (geom);
"""))
print(sql('INSERT INTO typed_parcels (inspire_id, geom) SELECT "INSPIREID"::bigint, geom FROM bham_parcels'))
print(sql("SELECT count(*), round((sum(area_m2) / 1e6)::numeric, 1)::float AS km2 FROM typed_parcels"))
ok
ok
ok
[(338611, 210.4)]
Four things in that table have no equivalent in a shapefile: a NOT NULL constraint, a UNIQUE key, a generated column that cannot drift from the geometry, and a declared geometry type and CRS that the database enforces on every insert.
The geometry(Polygon, 27700) declaration is the part people underuse. It means the database refuses bad rows at write time:
print(sql("INSERT INTO typed_parcels (inspire_id, geom) SELECT 1, ST_Multi(geom) FROM bham_parcels LIMIT 1"))
print(sql("INSERT INTO typed_parcels (inspire_id, geom) SELECT 2, ST_Transform(geom, 4326) FROM bham_parcels LIMIT 1"))
print(sql("INSERT INTO typed_parcels (inspire_id, geom) VALUES (3, NULL)"))
print(sql("UPDATE typed_parcels SET area_m2 = 0 WHERE parcel_id = 1"))
ERROR: Geometry type (MultiPolygon) does not match column type (Polygon)
ERROR: Geometry SRID (4326) does not match column SRID (27700)
ERROR: null value in column "geom" of relation "typed_parcels" violates not-null constraint
ERROR: column "area_m2" can only be updated to DEFAULT
Much of the data-cleaning work in a file-based pipeline is catching problems a typed column would have refused.
The spatial index is the performance story
query = """
EXPLAIN ANALYZE
SELECT count(*) FROM bham_parcels p JOIN bham_wards w ON ST_Intersects(p.geom, w.geom)
WHERE w."WD24NM" = 'Ladywood'
"""
def plan_summary(rows):
lines = [r[0] for r in rows]
scans = [l.strip().lstrip("-> ").split(" (")[0] for l in lines if "Scan" in l and "bham_parcels" in l]
return scans[0], [l for l in lines if l.startswith("Execution Time")][0]
print("with the GIST index: ", plan_summary(sql(query)))
with engine.begin() as conn: # SET lasts for this transaction only
conn.exec_driver_sql("SET LOCAL enable_indexscan = off")
conn.exec_driver_sql("SET LOCAL enable_bitmapscan = off")
print("index scans disabled: ", plan_summary(conn.exec_driver_sql(query).all()))
with the GIST index: ('Index Scan using idx_bham_parcels_geom on bham_parcels p', 'Execution Time: 9.500 ms')
index scans disabled: ('Parallel Seq Scan on bham_parcels p', 'Execution Time: 474.973 ms')
The index lookup tests bounding boxes first and runs the exact ST_Intersects only on the survivors; without it, every parcel in the city is tested against the ward. This is the same R-tree idea GeoPandas uses via sindex, with one difference that matters: the database index is persistent. GeoPandas builds one per session, on data it has already loaded into memory; PostGIS keeps one on disk and uses it without loading the table.
Where GeoPandas fits after the move
You do not stop using GeoPandas. You change what it receives.
from sklearn.cluster import KMeans
# PostGIS does the selection; GeoPandas does the analysis
gdf = gpd.read_postgis(
"""
SELECT p."INSPIREID", w."WD24NM" AS ward, p.geom
FROM bham_parcels p JOIN bham_wards w ON ST_Within(p.geom, w.geom)
WHERE w."WD24NM" = %(ward)s AND ST_Area(p.geom) > %(min_area)s
""",
engine, geom_col="geom", params={"ward": "Ladywood", "min_area": 1000},
)
# now do the thing SQL is bad at
xy = gdf.geometry.centroid.get_coordinates()
gdf["cluster"] = KMeans(n_clusters=5, n_init=10, random_state=0).fit_predict(xy)
print(len(gdf), "parcels; cluster sizes", sorted(gdf["cluster"].value_counts().tolist(), reverse=True))
466 parcels; cluster sizes [126, 121, 111, 57, 51]
The division that works: SQL for selection, joining and aggregation; Python for modelling, iteration and anything with a library behind it. Trying to do k-means in SQL is as unpleasant as trying to join tables that do not fit in memory in pandas.
What you give up
- A server to run. Someone has to back it up, upgrade it and hold the credentials.
- Portability. You cannot email a database. Exporting to GeoPackage for delivery becomes a pipeline step.
- Simplicity in version control. A shapefile has a checksum. A database has a migration history, which is better, but only once you set it up.
- Offline work. Files work on a train.
Loading Birmingham's parcels once and querying them
The data used on this page
HM Land Registry's INSPIRE Index Polygons for Birmingham (338,611 freehold parcels, Open Government Licence, Β© Crown copyright and database rights); Birmingham's 69 wards from the ONS Wards (December 2024) generalised boundaries; and buildings and schools from Geofabrik's OpenStreetMap extract of the West Midlands dated 1 January 2026 (Β© OpenStreetMap contributors, ODbL). They are saved as GeoPackages in data/, then loaded into a throwaway PostGIS 3.4 in Docker on port 55433. The helper sql() runs statements and returns rows β or the error message, so the errors on this page are the ones PostGIS gives.
import io
import os
import time
import zipfile
from pathlib import Path
import geopandas as gpd
import pandas as pd
import requests
from sqlalchemy import create_engine
from sqlalchemy.exc import DBAPIError
Path("data").mkdir(exist_ok=True)
if not Path("data/parcels.gpkg").exists():
reply = requests.Session().get(
"https://use-land-property-data.service.gov.uk/datasets/inspire/download/Birmingham.zip", timeout=600
)
reply.raise_for_status()
zipfile.ZipFile(io.BytesIO(reply.content)).extract("Land_Registry_Cadastral_Parcels.gml", "data")
gml = gpd.read_file("data/Land_Registry_Cadastral_Parcels.gml", columns=["INSPIREID"])
gml.to_file("data/parcels.gpkg")
os.remove("data/Land_Registry_Cadastral_Parcels.gml")
if not Path("data/wards.gpkg").exists():
gpd.read_file(
"https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/"
"Wards_December_2024_Boundaries_UK_BGC/FeatureServer/0/query"
"?where=LAD24NM%3D%27Birmingham%27&outFields=WD24CD,WD24NM&outSR=27700&f=geojson"
).to_file("data/wards.gpkg")
if not Path("data/buildings.gpkg").exists():
osm = Path("data/west-midlands-260101-free.shp.zip")
if not osm.exists():
osm.write_bytes(requests.get(
"https://download.geofabrik.de/europe/united-kingdom/england/west-midlands-260101-free.shp.zip",
timeout=600).content)
city = gpd.read_file("data/wards.gpkg").union_all()
box = tuple(gpd.GeoSeries([city], crs=27700).to_crs(4326).total_bounds)
layer = lambda name, **kw: gpd.read_file(f"/vsizip/{osm}/{name}", bbox=box, **kw).to_crs(27700)
buildings = layer("gis_osm_buildings_a_free_1.shp", columns=["osm_id", "type"])
buildings[buildings.intersects(city)].to_file("data/buildings.gpkg")
schools = pd.concat([
layer(name, columns=["osm_id", "fclass", "name"], where="fclass = 'school'")
for name in ("gis_osm_pois_free_1.shp", "gis_osm_pois_a_free_1.shp")
])
schools[schools.intersects(city)].to_file("data/schools.gpkg")
os.environ["PGPASSWORD"] = "guide"
if os.system("docker inspect guide-postgis >/dev/null 2>&1") != 0:
os.system("docker run -d --name guide-postgis -e POSTGRES_USER=user -e POSTGRES_PASSWORD=guide "
"-e POSTGRES_DB=gis -p 127.0.0.1:55433:5432 postgis/postgis:16-3.4 >/dev/null")
time.sleep(10)
engine = create_engine("postgresql+psycopg://user@localhost:55433/gis")
def sql(statement):
try:
with engine.begin() as conn:
result = conn.exec_driver_sql(statement)
return result.all() if result.returns_rows else "ok"
except DBAPIError as err:
return "ERROR: " + str(err.orig).splitlines()[0]
for name in ("parcels", "wards", "buildings", "schools"):
layer = gpd.read_file(f"data/{name}.gpkg").rename_geometry("geom")
layer.to_postgis(f"bham_{name}", engine, if_exists="replace", index=False)
print(f"bham_{name}: {len(layer):,} rows")
sql("ANALYZE bham_parcels; ANALYZE bham_wards; ANALYZE bham_buildings; ANALYZE bham_schools")
bham_parcels: 338,611 rows
bham_wards: 69 rows
bham_buildings: 257,112 rows
bham_schools: 491 rows
Example 1: loading a folder of files into PostGIS once
The setup already loaded four GeoPackages the usual way. What to_postgis leaves behind is worth checking rather than assuming:
from pathlib import Path
for path in sorted(Path("data").glob("*.gpkg")):
print(f"{path.name:16s} {len(gpd.read_file(path, columns=[])):>8,} rows")
print(sql("SELECT tablename, indexname FROM pg_indexes WHERE left(tablename, 5) = 'bham_' ORDER BY 1"))
buildings.gpkg 257,112 rows
parcels.gpkg 338,611 rows
schools.gpkg 491 rows
wards.gpkg 69 rows
[('bham_buildings', 'idx_bham_buildings_geom'), ('bham_parcels', 'idx_bham_parcels_geom'), ('bham_schools', 'idx_bham_schools_geom'), ('bham_wards', 'idx_bham_wards_geom')]
to_postgis creates the GIST index on the geometry column itself (GeoAlchemy2 adds it when it creates the table). What it does not do is keep anything else: with if_exists="replace" the table is dropped and rebuilt, and any index or constraint you added by hand goes with it.
wards = gpd.read_file("data/wards.gpkg").rename_geometry("geom")
wards.to_postgis("index_demo", engine, if_exists="replace", index=False)
sql('CREATE INDEX index_demo_name ON index_demo ("WD24NM")')
print("before:", sql("SELECT indexname FROM pg_indexes WHERE tablename = 'index_demo' ORDER BY 1"))
wards.to_postgis("index_demo", engine, if_exists="replace", index=False)
print("after replace:", sql("SELECT indexname FROM pg_indexes WHERE tablename = 'index_demo' ORDER BY 1"))
sql("DROP TABLE index_demo")
before: [('idx_index_demo_geom',), ('index_demo_name',)]
after replace: [('idx_index_demo_geom',)]
Then run ANALYZE, so the planner has statistics. Here is a freshly created copy of the parcels with autovacuum switched off, queried for a 1 km square before and after:
with engine.begin() as conn:
conn.exec_driver_sql("DROP TABLE IF EXISTS fresh")
conn.exec_driver_sql("CREATE TABLE fresh WITH (autovacuum_enabled = false) AS SELECT * FROM bham_parcels")
conn.exec_driver_sql("CREATE INDEX ON fresh USING gist (geom)")
box = "SELECT count(*) FROM fresh WHERE geom && ST_MakeEnvelope(406000, 286000, 407000, 287000, 27700)"
def estimate():
line = next(r[0] for r in sql("EXPLAIN ANALYZE " + box) if "Heap Scan" in r[0] or "Seq Scan" in r[0])
rows = line.split("rows=")
return line.split(" (")[0].strip("-> "), "estimated", rows[1].split()[0], "actual", rows[2].split()[0]
print("before ANALYZE:", estimate())
sql("ANALYZE fresh")
print("after ANALYZE: ", estimate())
sql("DROP TABLE fresh")
before ANALYZE: ('Bitmap Heap Scan on fresh', 'estimated', '34', 'actual', '553')
after ANALYZE: ('Bitmap Heap Scan on fresh', 'estimated', '553', 'actual', '553')
Without statistics PostgreSQL still used the index for this small query, but its row estimate was well out; after ANALYZE it was close. Estimates that far out do not matter for one table, and decide join order once several tables are involved. Autovacuum analyses a new table on its own after a short delay, but a script that loads and then immediately queries should not wait for it.
Example 2: the query that files cannot answer cheaply
Parcels of more than a hectare with no building on them, within 200 m of a school, with the ward each one is in:
query = """
SELECT p."INSPIREID", w."WD24NM" AS ward, round(ST_Area(p.geom)) AS area_m2
FROM bham_parcels p
JOIN bham_wards w ON ST_Within(ST_PointOnSurface(p.geom), w.geom)
WHERE ST_Area(p.geom) > 10000
AND NOT EXISTS (SELECT 1 FROM bham_buildings b WHERE ST_Intersects(b.geom, p.geom))
AND EXISTS (SELECT 1 FROM bham_schools s WHERE ST_DWithin(s.geom, p.geom, 200))
ORDER BY area_m2 DESC
"""
start = time.perf_counter()
in_sql = pd.read_sql(query, engine)
print(f"PostGIS: {len(in_sql)} parcels in {time.perf_counter() - start:.2f} s")
print(in_sql.head(5).to_string(index=False))
# the same question from the files
start = time.perf_counter()
parcels = gpd.read_file("data/parcels.gpkg")
buildings = gpd.read_file("data/buildings.gpkg")
schools = gpd.read_file("data/schools.gpkg")
ward_layer = gpd.read_file("data/wards.gpkg")
loaded = time.perf_counter()
big = parcels[parcels.area > 10_000]
built_on = gpd.sjoin(big, buildings[["geometry"]], predicate="intersects").index.unique()
empty = big.drop(built_on)
near = gpd.sjoin(empty, schools[["geometry"]], predicate="dwithin", distance=200).index.unique()
found = empty.loc[near]
found = gpd.sjoin(found.set_geometry(found.representative_point()), ward_layer[["WD24NM", "geometry"]], predicate="within")
print(f"GeoPandas: {len(found)} parcels; {loaded - start:.2f} s loading, {time.perf_counter() - loaded:.2f} s joining")
print("same parcels:", sorted(found["INSPIREID"]) == sorted(in_sql["INSPIREID"]))
PostGIS: 120 parcels in 0.23 s
INSPIREID ward area_m2
61656971 Sutton Walmley & Minworth 481373.0
24769507 Sutton Walmley & Minworth 430791.0
23346692 Bartley Green 203265.0
24720347 Sutton Walmley & Minworth 188295.0
60031465 Ward End 167782.0
GeoPandas: 120 parcels; 0.83 s loading, 0.21 s joining
same parcels: True
Both give the same answer. At this size GeoPandas is not slow β most of its time goes on reading four files into memory, and the joins themselves take about as long as the whole SQL query. The difference is what each one needs: the SQL touches four indexes and returns a few rows to a client that holds nothing else, and it will do the same when the tables are ten times bigger than the machine's memory. The GeoPandas version has to hold every building in the city before it can rule a single parcel out.
Example 3: keeping files in the workflow anyway
# database is the source of truth; files are the deliverable
def export_ward(ward: str, out_dir: Path) -> Path:
gdf = gpd.read_postgis(
"""SELECT p."INSPIREID", p.geom FROM bham_parcels p
JOIN bham_wards w ON ST_Within(ST_PointOnSurface(p.geom), w.geom)
WHERE w."WD24NM" = %(w)s""",
engine, geom_col="geom", params={"w": ward},
)
out_dir.mkdir(exist_ok=True)
out = out_dir / f"{ward.lower().replace(' ', '_')}_parcels.gpkg"
gdf.to_file(out, driver="GPKG", layer="parcels")
return out
out = export_ward("Ladywood", Path("output"))
print(out, len(gpd.read_file(out)), "parcels")
output/ladywood_parcels.gpkg 2582 parcels
This is the pattern most teams land on: PostGIS holds the canonical data, and every delivery is a generated GeoPackage. Nobody edits the deliverable, because regenerating it is one function call.
Where the filtering happens
The deep difference is where the filtering happens.
A file format is a container. To find the large parcels in Ladywood from a plain read_file(), something must read every parcel in Birmingham, because there is nowhere else the information could be. GeoPackage helps β it is SQLite underneath and carries an R-tree index, and mask= or bbox= uses it β but that is one spatial filter on one layer. A join across four layers still means loading all four.
A database is a query engine wrapped around storage. It has statistics about the data, indexes over it, and a planner that decides how to combine them. It answers questions without materialising the data in the client, and it does that on data far larger than the memory of the machine asking.
That is also why the measured gap on this page was small for one filter and large in kind for the four-table query: the GeoPackage's R-tree served the Ladywood filter as well as PostGIS did, but only the database could rule parcels out against buildings, schools and wards without the client holding all four. The other differences β transactions for concurrent writers, and typed columns that refused a bad row above β are about managing data rather than querying it, and they are usually what tips the decision.
None of that makes files obsolete. The right architecture for most teams is a database as the system of record and files as the interchange format, which is exactly how the rest of the data world works.
GeoPackage, SRIDs and replace
- GeoPackage is a database too. If you need indexes and SQL but not concurrency or a server,
GPKGplusmask=/where=orsqlite3gets you a surprising distance for free β the Ladywood filter above ran about as fast from the file as from PostGIS. - One CRS per table, chosen deliberately. An untyped
geometrycolumn, orgeometry(Geometry, 0), accepts any SRID β a test table here happily held parcels in 27700 and 4326 side by side β and is a permanent source of confusion. Reproject on the way in and declare the SRID. to_postgis(if_exists="replace")drops the table. It recreates the geometry index but not your other indexes or constraints. Useif_exists="append"after the initial load, or recreate them afterwards.
Internal links
- How to connect GeoPandas to PostGIS β the connection string, drivers and first query
- How to write a GeoDataFrame to PostGIS β loading data in without losing types or CRS
- How to run spatial SQL queries from Python with PostGIS β pushing the work down to the database
- PostGIS write fails on SRID or geometry type β the errors a typed column produces, and why they are useful
- How to load and write PostGIS layers from PyQGIS β the same database from QGIS
- GIS vector file formats compared β what you are moving away from, and what it does well
- Spatial indexes explained: R-trees and why spatial joins are fast β the idea the GIST index implements
- How to process a very large GeoPackage in chunks β the file-based alternative when a server is not an option
FAQ
Is PostGIS faster than GeoPandas?
For selection it is much faster than reading a whole file and filtering in pandas: the Ladywood query above returned in hundredths of a second where reading all 338,611 Birmingham parcels first took over half a second. But a GeoPackage read with mask= was about as quick, and on a four-table join GeoPandas' own joins took about as long as the whole SQL query once the files were loaded. The database pulls ahead as the data outgrows memory and as more tables join. For row-wise computation on data already in memory, GeoPandas is faster, because there is no network in between.
Can I use PostGIS without writing SQL?
Partly. GeoPandas can read a whole table by name and write with to_postgis. But the reason to use a database is to push work into it, and that means SQL β at least WHERE clauses and joins.
How much data justifies a database?
Size is the weakest of the five signals. Ten thousand rows edited by three people justifies a database; ten million read-only rows queried by one script may not.
Can PostGIS handle rasters?
There is a postgis_raster extension, but most teams keep rasters as Cloud-Optimised GeoTIFFs on object storage and reference them from the database. Vector in the database, raster beside it, is the common split.