What GDAL and OGR Actually Are (and Why Everything Depends on Them)
On this page
Problem statement
You install GeoPandas and hit an error that mentions a library you did not install:
ImportError: libgdal.so.34: cannot open shared object file: No such file or directoryrasterio.errors.RasterioIOError: data/raw/dem.tif: No such file or directoryERROR 1: PROJ: internal_proj_create_from_database: Cannot find proj.dbThe last two are real, from rasterio 1.5 on GDAL 3.12 — the second after pointing PROJ_DATA at a directory that does not exist (Example 5).
CPLE, PROJ, libgdal — none of these are Python. Underneath GeoPandas, Fiona, pyogrio, rasterio, QGIS, PostGIS, ArcGIS and virtually every other GIS tool sits one C++ library: GDAL. Almost every confusing installation problem, driver limitation and format quirk in the Python GIS stack is a GDAL fact showing through.
Knowing what GDAL is, what OGR was, and how Python reaches it turns those messages from mysterious into diagnosable.
Quick answer
GDAL is the translation layer for geospatial data:
- GDAL = Geospatial Data Abstraction Library — one API over about 150 raster and 80 vector formats, depending on the build
- OGR was the separate vector half; since GDAL 2.0 they are one library, and the name survives in
ogr2ograndogrinfo - It ships drivers (one per format), a virtual filesystem (
/vsizip/,/vsis3/), and CRS support via PROJ - Python touches it through
pyogrioandfiona(vector) andrasterio(raster) — all of them wrappers - Most "GeoPandas" limits are GDAL driver limits, and most install failures are GDAL library mismatches
import geopandas as gpd
import pyogrio, rasterio
print("GDAL (via rasterio):", rasterio.__gdal_version__)
print("GDAL (via pyogrio) :", pyogrio.__gdal_version_string__)
print("PROJ :", rasterio.__proj_version__)
print("vector drivers :", len(pyogrio.list_drivers()))
print("can write GPKG :", pyogrio.list_drivers(write=True).get("GPKG"))GDAL (via rasterio): 3.12.4
GDAL (via pyogrio) : 3.12.4
PROJ : 9.8.1
vector drivers : 64
can write GPKG : raw
If those numbers disagree between libraries, you have more than one GDAL build in the environment. That alone is normal: each pip wheel carries a private copy under a renamed library file, and in this environment four copies — rasterio's, pyogrio's, Fiona's and the osgeo bindings' — load side by side without trouble. What breaks is two builds that share a library name, typically a conda or system GDAL meeting another copy on the same search path.
What sits under GeoPandas
Drivers, virtual files, configuration and PROJ
gpd.read_file().One API, a hundred formats
GDAL's central idea is abstraction. Your code opens "a dataset"; GDAL works out which driver can read it and hands back a uniform object. That is why gpd.read_file() takes a shapefile, a GeoPackage, a GeoJSON, a File Geodatabase, a WFS URL or a PostGIS table with no change to your code.
import pyogrio
import pandas as pd
drivers = pyogrio.list_drivers()
print(f"{len(drivers)} vector drivers available")
table = pd.DataFrame(
[{"driver": k, "capability": v} for k, v in sorted(drivers.items())]
)
wanted = ["ESRI Shapefile", "GPKG", "GeoJSON", "FlatGeobuf", "Parquet", "PostgreSQL", "OpenFileGDB"]
print(table[table["driver"].isin(wanted)].to_string(index=False))
print("not in this build:", [d for d in wanted if d not in drivers])64 vector drivers available
driver capability
ESRI Shapefile raw
FlatGeobuf rw
GPKG raw
GeoJSON raw
OpenFileGDB raw
not in this build: ['Parquet', 'PostgreSQL']
"r" means read-only, "rw" read and write, "raw" read, append and write. A driver that is missing entirely is why a format "is not supported" — the format is fine, your GDAL build just was not compiled with it. The pyogrio wheel measured here has no Parquet and no PostgreSQL driver; the system GDAL on the same machine has both.
OGR: the vector half, and why the name persists
GDAL began in 1998 as a raster library. OGR ("OpenGIS Simple Features Reference Implementation") arrived as a separate vector library, and the two merged in GDAL 2.0 in 2015. The names survive in the tooling:
gdalinfo data/raw/dem.tif # raster metadata
ogrinfo -so -al data/raw/places.gpkg # vector metadata
gdal_translate -of COG in.tif out.tif # raster conversion
ogr2ogr -f GPKG out.gpkg in.shp # vector conversionIn Python, the split shows up as from osgeo import gdal, ogr, osr — three modules of one library. Modern code rarely uses them directly, because rasterio and pyogrio are far more pleasant, but they are the same functionality.
Drivers: where the format limits live
Every quirk you have met is a driver behaviour:
import pyogrio
for name in ["ESRI Shapefile", "GPKG", "GeoJSON"]:
print(f"{name:16} {pyogrio.list_drivers().get(name)}")ESRI Shapefile raw
GPKG raw
GeoJSON raw
The shapefile driver truncates field names to ten bytes because dBase does. The GeoJSON driver writes whatever CRS it is given, with a legacy crs member, unless you pass RFC7946=YES — then it reprojects to WGS 84 because RFC 7946 says so. The GPKG driver supports multiple layers because SQLite does. None of this is GeoPandas — GeoPandas simply passes your data to a driver and reports what happens.
Driver behaviour is configurable through open and creation options:
import geopandas as gpd
gdf = gpd.read_file("data/raw/places.gpkg").to_crs(27700)
# creation options are passed through to the driver
gdf.to_file("out/places.geojson", driver="GeoJSON",
COORDINATE_PRECISION=6, RFC7946="YES")
gdf.to_file("out/places.shp", driver="ESRI Shapefile", encoding="utf-8")
print(gpd.read_file("out/places.geojson").crs, "| shapefile columns:",
[c for c in gpd.read_file("out/places.shp").columns if len(c) >= 10][:3])EPSG:4326 | shapefile columns: ['population']
The GeoDataFrame was in British National Grid; RFC7946=YES made the driver write WGS 84, and the shapefile driver cut long column names to ten characters.
The virtual filesystem: reading things that are not files
One of GDAL's most useful and least known features is /vsi*/, a set of virtual filesystem handlers that make archives, memory and cloud storage look like paths.
import geopandas as gpd
# inside a zip, without unpacking
print(len(gpd.read_file("/vsizip/data/raw/places.zip/ne_110m_populated_places.shp")), "from a zip")
# straight from a web server (or /vsis3/ for object storage)
print(len(gpd.read_file("/vsicurl/https://flatgeobuf.org/test/data/UScounties.fgb",
bbox=(-105.3, 39.6, -104.6, 40.1))), "counties from a remote FlatGeobuf")
# gzip, tar and nesting all work
print(len(gpd.read_file("/vsigzip/data/raw/places.geojson.gz")), "from gzip")
print(len(gpd.read_file("/vsizip//vsicurl/https://naciscdn.org/naturalearth/110m/cultural/"
"ne_110m_populated_places.zip/ne_110m_populated_places.shp")), "from a remote zip")243 from a zip
7 counties from a remote FlatGeobuf
243 from gzip
243 from a remote zip
Combined with a format that supports partial reads, this is how a bbox query against a remote FlatGeobuf transfers only the bytes it needs: the seven counties around Denver came from a 14.1 MB file in six range requests totalling 589,824 bytes, about 4% of it (Example 2).
Configuration through environment variables
GDAL is configured by environment variables far more than by API calls, which is why containers and cron jobs behave differently from laptops:
import os
os.environ["GDAL_CACHEMAX"] = "512" # MB of block cache
os.environ["GDAL_NUM_THREADS"] = "ALL_CPUS" # threaded compression, warping
os.environ["GDAL_DISABLE_READDIR_ON_OPEN"] = "EMPTY_DIR" # essential for /vsis3 speed
os.environ["CPL_DEBUG"] = "ON" # verbose driver logging
os.environ["OGR_GEOMETRY_ACCEPT_UNCLOSED_RING"] = "NO"CPL_DEBUG=ON is the diagnostic to remember: CPL is GDAL's portability layer, and turning it on prints exactly which driver was tried, which file was opened and why something failed.
PROJ: the other library underneath
Coordinate transformations are not GDAL's own work; they are PROJ's. That separation explains a whole family of errors:
import pyproj
print("PROJ version:", pyproj.proj_version_str)
print("PROJ data dir:", pyproj.datadir.get_data_dir().replace(os.path.expanduser("~"), "~"))
print(pyproj.CRS.from_epsg(27700).name)PROJ version: 9.8.1
PROJ data dir: ~/WebstormProjects/spatialworkflow_11ty/.cache/gisenv/lib/python3.14/site-packages/pyproj/proj_dir/share/proj
OSGB36 / British National Grid
ERROR 1: PROJ: internal_proj_create_from_database: Cannot find proj.dbThat message means PROJ cannot find its coordinate-operation database — usually because PROJ_LIB/PROJ_DATA points somewhere wrong, or because two PROJ installations are fighting. It is never a problem with your data. Each wheel carries its own proj.db, and they do not all react the same way: with PROJ_DATA pointed at nothing, rasterio and the osgeo bindings failed with this message while pyproj carried on with its bundled copy (Example 5).
Why installations break
import rasterio, pyogrio, fiona
print("rasterio's GDAL:", rasterio.__gdal_version__)
print("pyogrio's GDAL :", pyogrio.__gdal_version_string__)
print("fiona's GDAL :", fiona.__gdal_version__)rasterio's GDAL: 3.12.4
pyogrio's GDAL : 3.12.4
fiona's GDAL : 3.12.2
Three numbers that need not match — here two say 3.12.4 and one 3.12.2, and everything works. They diverge because there are two ways to get GDAL into a Python environment:
- wheels —
pip install rasteriodownloads a wheel with its own GDAL bundled inside - system library — conda,
aptor a Docker base image provides one shared GDAL
Wheels rename their bundled library (libgdal-253d08a6.so…), so several can share a process, as four do here. Trouble starts when a conda or system GDAL of the same name joins them; how to build an environment that avoids it is setting up a Python GIS environment.
Inspecting and driving GDAL from Python
The data used on this page
Natural Earth's 243 populated places, kept three ways — as the original zip, as a GeoPackage and gzipped GeoJSON — plus two remote files read in place: the FlatGeobuf project's 14 MB file of US counties and the Natural Earth zip on its download server.
import gzip
import os
import geopandas as gpd
import requests
os.makedirs("data/raw", exist_ok=True)
os.makedirs("data/out", exist_ok=True)
os.makedirs("out", exist_ok=True)
if not os.path.exists("data/raw/places.zip"):
open("data/raw/places.zip", "wb").write(requests.get(
"https://naciscdn.org/naturalearth/110m/cultural/ne_110m_populated_places.zip", timeout=120,
headers={"User-Agent": "spatialworkflow.io guide"}).content)
places = gpd.read_file("data/raw/places.zip")[["NAME", "ADM0NAME", "POP_MAX", "geometry"]]
places = places.rename(columns={"POP_MAX": "population_max"})
places.to_file("data/raw/places.gpkg")
places.to_file("data/raw/places.geojson")
open("data/raw/places.geojson.gz", "wb").write(gzip.compress(open("data/raw/places.geojson", "rb").read()))
print(sorted(os.listdir("data/raw")))['places.geojson', 'places.geojson.gz', 'places.gpkg', 'places.zip']
Example 1: an environment report worth keeping
"""gdal_env.py — print everything needed to diagnose a GIS environment."""
import sys, os
HOME = os.path.expanduser("~")
show = lambda path: str(path).replace(HOME, "~") # keep the report shareable
def report():
print("python :", sys.version.split()[0], "|", show(sys.executable))
try:
from osgeo import gdal
print("gdal (osgeo) :", gdal.__version__, "|", show(gdal.__file__))
except ImportError:
print("gdal (osgeo) : not installed (fine — wrappers do not need it)")
for module, attr in [("rasterio", "__gdal_version__"),
("pyogrio", "__gdal_version_string__"),
("fiona", "__gdal_version__")]:
try:
mod = __import__(module)
print(f"{module:14}: GDAL {getattr(mod, attr)} | {show(mod.__file__)}")
except ImportError:
print(f"{module:14}: not installed")
import pyproj
print("proj :", pyproj.proj_version_str, "|", show(pyproj.datadir.get_data_dir()))
for var in ["GDAL_DATA", "PROJ_LIB", "PROJ_DATA", "GDAL_CACHEMAX", "GDAL_NUM_THREADS"]:
print(f" ${var:16} = {os.environ.get(var, '<unset>')}")
import pyogrio
drivers = pyogrio.list_drivers()
print(f"vector drivers: {len(drivers)}")
for key in ["ESRI Shapefile", "GPKG", "GeoJSON", "FlatGeobuf", "Parquet", "PostgreSQL"]:
print(f" {key:16} {drivers.get(key, 'MISSING')}")
report()python : 3.14.4 | ~/WebstormProjects/spatialworkflow_11ty/.cache/gisenv/bin/python
gdal (osgeo) : 3.12.2 | ~/WebstormProjects/spatialworkflow_11ty/.cache/gisenv/lib/python3.14/site-packages/osgeo/gdal.py
rasterio : GDAL 3.12.4 | ~/WebstormProjects/spatialworkflow_11ty/.cache/gisenv/lib/python3.14/site-packages/rasterio/__init__.py
pyogrio : GDAL 3.12.4 | ~/WebstormProjects/spatialworkflow_11ty/.cache/gisenv/lib/python3.14/site-packages/pyogrio/__init__.py
fiona : GDAL 3.12.2 | ~/WebstormProjects/spatialworkflow_11ty/.cache/gisenv/lib/python3.14/site-packages/fiona/__init__.py
proj : 9.8.1 | ~/WebstormProjects/spatialworkflow_11ty/.cache/gisenv/lib/python3.14/site-packages/pyproj/proj_dir/share/proj
$GDAL_DATA = <unset>
$PROJ_LIB = <unset>
$PROJ_DATA = <unset>
$GDAL_CACHEMAX = <unset>
$GDAL_NUM_THREADS = <unset>
vector drivers: 64
ESRI Shapefile raw
GPKG raw
GeoJSON raw
FlatGeobuf rw
Parquet MISSING
PostgreSQL MISSING
Run this first whenever a GIS environment misbehaves — it answers most questions before you start guessing.
Example 2: use the virtual filesystem instead of downloading
import geopandas as gpd
import os
os.environ["GDAL_DISABLE_READDIR_ON_OPEN"] = "EMPTY_DIR" # avoid listing the whole bucket
os.environ["VSI_CACHE"] = "TRUE"
# read a bbox from a remote FlatGeobuf: only the needed byte ranges are fetched
url = "/vsicurl/https://flatgeobuf.org/test/data/UScounties.fgb"
subset = gpd.read_file(url, bbox=(-105.3, 39.6, -104.6, 40.1))
print(f"{len(subset):,} counties without downloading the file:", sorted(subset["NAME"])[:4], "…")
# and a shapefile inside a zip inside an HTTP server
nested = gpd.read_file("/vsizip//vsicurl/https://naciscdn.org/naturalearth/110m/cultural/"
"ne_110m_populated_places.zip/ne_110m_populated_places.shp")
print(len(nested), "places from inside a remote zip")
# how many bytes did the bbox read actually move? GDAL can say, in a fresh process
import subprocess, sys, json
probe = f"""import geopandas as gpd
gpd.read_file("{url}", bbox=(-105.3, 39.6, -104.6, 40.1))"""
run = subprocess.run([sys.executable, "-c", probe], capture_output=True, text=True,
env=dict(os.environ, CPL_VSIL_SHOW_NETWORK_STATS="YES"))
output = run.stdout + run.stderr # GDAL prints the statistics as JSON at exit
stats = json.loads(output[output.index("{"):])["methods"]["GET"]
print(f"{stats['count']} range requests, {stats['downloaded_bytes']:,} bytes of a 14,100,008-byte file")7 counties without downloading the file: ['Adams', 'Arapahoe', 'Boulder', 'Broomfield'] …
243 places from inside a remote zip
6 range requests, 589,824 bytes of a 14,100,008-byte file
Example 3: driving ogr2ogr from Python for bulk work
import subprocess
from pathlib import Path
def ogr2ogr(src, dest, *, layer=None, where=None, t_srs=None, fmt="GPKG", extra=()):
cmd = ["ogr2ogr", "-f", fmt, str(dest), str(src)]
if layer: cmd += [layer]
if where: cmd += ["-where", where]
if t_srs: cmd += ["-t_srs", t_srs]
cmd += ["-progress", *extra]
proc = subprocess.run(cmd, capture_output=True, text=True)
if proc.returncode != 0:
raise RuntimeError(f"ogr2ogr failed: {proc.stderr.strip()[-500:]}")
return Path(dest)
out = ogr2ogr("data/raw/places.gpkg", "data/out/uk_places.gpkg",
where="ADM0NAME = 'United Kingdom'", t_srs="EPSG:27700")
print(subprocess.run(["ogrinfo", "--version"], capture_output=True, text=True).stdout.strip())
print(gpd.read_file(out)[["NAME"]].to_string(index=False), gpd.read_file(out).crs)GDAL 3.12.2 "Chicoutimi", released 2026/02/03
NAME
London EPSG:27700
The ogr2ogr on this machine is the system's GDAL 3.12.2, not the 3.12.4 inside the Python wheels — which is exactly the "CLI and Python use different GDALs" situation in the notes below.
For a straight format conversion or a bulk load of millions of features, ogr2ogr is usually faster than reading into Python and writing back out — it never builds the objects.
Example 4: capture GDAL's own errors properly
import logging
from osgeo import gdal
gdal.UseExceptions() # turn silent error codes into Python exceptions
logging.basicConfig(level=logging.INFO)
log = logging.getLogger("gdal")
def gdal_error_handler(err_class, err_num, message):
level = {gdal.CE_Warning: log.warning, gdal.CE_Failure: log.error,
gdal.CE_Fatal: log.critical}.get(err_class, log.info)
level("GDAL %s: %s", err_num, message.strip())
gdal.PushErrorHandler(gdal_error_handler)
try:
ds = gdal.OpenEx("data/raw/missing.gpkg", gdal.OF_VECTOR)
except RuntimeError as exc:
print("open failed:", exc)
finally:
gdal.PopErrorHandler()open failed: data/raw/missing.gpkg: No such file or directory
Example 5: a wrong PROJ_DATA, three libraries
import subprocess, sys
CHECKS = {
"rasterio": "import rasterio.crs; print(rasterio.crs.CRS.from_epsg(27700).to_epsg())",
"osgeo": "from osgeo import osr; osr.UseExceptions(); s = osr.SpatialReference(); "
"s.ImportFromEPSG(27700); print(s.GetName())",
"pyproj": "import pyproj; print(pyproj.CRS.from_epsg(27700).name)",
}
for name, code in CHECKS.items():
run = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True,
env=dict(os.environ, PROJ_DATA="/nonexistent", PROJ_LIB="/nonexistent"))
last = (run.stdout.strip() or run.stderr.strip().splitlines()[-1])[:90]
print(f"{name:8} {last}")rasterio rasterio.errors.CRSError: The EPSG code is unknown. PROJ: internal_proj_create_from_databa
osgeo RuntimeError: PROJ: proj_create_from_database: Cannot find proj.db
pyproj OSGB36 / British National Grid
gdal.UseExceptions() is worth setting in any script that touches the raw bindings: without it, GDAL signals failure by returning None and setting an error code, which is very easy to ignore.
Why one library reads everything
GDAL exists because the alternative is unthinkable. Without it, every GIS application would need its own reader for shapefile, GeoPackage, GeoJSON, KML, GML, MapInfo TAB, File Geodatabase, PostGIS, GeoTIFF, JPEG2000, NetCDF, HDF and a hundred more — each with its own bugs and its own gaps. Frank Warmerdam started it in 1998 for exactly this reason, and the result is that essentially the entire industry, open source and proprietary alike, reads data through one library.
The abstraction has a shape worth understanding. A driver knows one format. A dataset is an open file or connection. A vector dataset has layers, each with a feature definition (the field schema and geometry type) and features. A raster dataset has bands, a geotransform and a CRS. Every Python wrapper you use is presenting some part of that model — pyogrio.read_info() is a feature-definition query; rasterio's profile is the raster dataset's metadata.
Because the model is uniform but the formats are not, GDAL has to decide what to do when a format cannot express something. Those decisions are the driver behaviours that surface as quirks: truncated field names, dropped time components, reprojected GeoJSON, a boolean stored as an integer. When something surprising happens on write, the question to ask is "what did this driver do?", and the answer is in GDAL's driver documentation rather than in GeoPandas'.
PROJ is the other foundation, and keeping the two straight prevents a lot of confusion. GDAL reads and writes; PROJ transforms coordinates and holds the EPSG database. A CRS error mentioning proj.db is a PROJ installation problem. A driver error mentioning CPLE_OpenFailed is a GDAL one. Both can be triggered by the same broken environment, but they have different fixes.
Builds, defaults and GDAL 4
gdal.UseExceptions()will be the default: GDAL 4 makes exceptions mandatory. Set it now to avoid silentNonereturns.- Driver availability varies by build: conda-forge,
apt, wheels and the QGIS bundle all compile different driver sets. Check withpyogrio.list_drivers()rather than assuming. /vsicurl/needs a range-capable server: Without HTTP range support it downloads the whole file.ogr2ogrand Python may use different GDALs: The CLI comes from the system; the Python wrapper may bundle its own. Compare versions when results differ.- The bindings are not the wrappers:
osgeo.ogris the raw SWIG binding;pyogrio/fionaare friendlier. Mixing them in one script is legal but confusing. - GDAL is thread-safe with care: Dataset objects are not shareable between threads. Open per thread, or use processes.
Internal links
- Fiona vs pyogrio: How GeoPandas Reads and Writes Files
- GeoPandas Installation Fails: How to Fix Common Errors
- How to Containerise a Python GIS Pipeline with Docker
- GIS Vector File Formats Compared: Shapefile, GeoPackage, GeoJSON, Parquet
- RasterioIOError: Not Recognized as a Supported File Format (How to Fix)
FAQ
What is the difference between GDAL and OGR?
Historically GDAL handled rasters and OGR handled vectors. They merged in GDAL 2.0 (2015); "OGR" now survives only in tool names like ogr2ogr and in the osgeo.ogr module.
Why does an error mention CPLE or CPL?
CPL is GDAL's portability layer, so CPLE_* errors come from GDAL itself — file opening, driver selection, HTTP access. Set CPL_DEBUG=ON to see the full driver trace.
What is /vsizip/ and /vsis3/?
GDAL's virtual filesystem handlers. They let you open files inside archives, in memory, or in cloud storage using ordinary path syntax, with no unpacking or downloading step.
Why do two libraries report different GDAL versions?
Because each bundled its own copy, or one is using the system library. Two GDALs in one process cause symbol and PROJ database errors — standardise on wheels or on conda, not both.
Is ogr2ogr faster than doing it in Python?
For pure format conversion or bulk loading, usually yes: it streams features through GDAL without ever constructing Python objects. Use Python when you need per-feature logic.
What does PROJ do that GDAL does not?
PROJ owns coordinate reference systems and transformations, including the EPSG database in proj.db. GDAL calls into it. That is why CRS errors and format errors have different causes and different fixes.