🧭 Concept Files & formats Getting started Batch Processing · step 3 of 26 Updated

What GDAL and OGR Actually Are (and Why Everything Depends on Them)

Problem statement

You install GeoPandas and hit an error that mentions a library you did not install:

ImportError: libgdal.so.34: cannot open shared object file: No such file or directory
rasterio._err.CPLE_OpenFailedError: data/raw/dem.tif: No such file or directory
ERROR 1: PROJ: proj_create_from_database: Cannot find proj.db

CPLE, PROJ, libgdal — none of these are Python. Underneath GeoPandas, Fiona, pyogrio, rasterio, QGIS, PostGIS, ArcGIS and virtually every other GIS tool sits one C++ library: GDAL. Almost every confusing installation problem, driver limitation and format quirk in the Python GIS stack is a GDAL fact showing through.

Knowing what GDAL is, what OGR was, and how Python reaches it turns those messages from mysterious into diagnosable.

Quick answer

GDAL is the translation layer for geospatial data:

  1. GDAL = Geospatial Data Abstraction Library — one API over ~150 raster and ~100 vector formats
  2. OGR was the separate vector half; since GDAL 2.0 they are one library, and the name survives in ogr2ogr and ogrinfo
  3. It ships drivers (one per format), a virtual filesystem (/vsizip/, /vsis3/), and CRS support via PROJ
  4. Python touches it through pyogrio and fiona (vector) and rasterio (raster) — all of them wrappers
  5. Most "GeoPandas" limits are GDAL driver limits, and most install failures are GDAL library mismatches
import geopandas as gpd
import pyogrio, rasterio

print("GDAL (via rasterio):", rasterio.__gdal_version__)
print("GDAL (via pyogrio) :", pyogrio.__gdal_version_string__)
print("PROJ               :", rasterio.__proj_version__)
print("vector drivers     :", len(pyogrio.list_drivers()))
print("can write GPKG     :", pyogrio.list_drivers(write=True).get("GPKG"))

If those numbers disagree between libraries, you have two GDAL builds in one environment — which is the root cause of a large share of Python GIS installation problems.

What sits under GeoPandas

Layered stack from GeoPandas through pyogrio and GDAL down to drivers and PROJ.
Five layers — Python is only the top one, and the errors come from the bottom.

Step-by-step solution

Flow of a read_file call from GeoPandas through pyogrio, GDAL, the driver and the file.
What actually happens when you call `gpd.read_file()`.

One API, a hundred formats

GDAL's central idea is abstraction. Your code opens "a dataset"; GDAL works out which driver can read it and hands back a uniform object. That is why gpd.read_file() takes a shapefile, a GeoPackage, a GeoJSON, a File Geodatabase, a WFS URL or a PostGIS table with no change to your code.

import pyogrio
import pandas as pd

drivers = pyogrio.list_drivers()
print(f"{len(drivers)} vector drivers available")

table = pd.DataFrame(
    [{"driver": k, "capability": v} for k, v in sorted(drivers.items())]
)
print(table[table["driver"].isin(
    ["ESRI Shapefile", "GPKG", "GeoJSON", "FlatGeobuf", "Parquet", "PostgreSQL", "OpenFileGDB"]
)].to_string(index=False))

"rw" means read and write, "r" read-only. A driver that is missing entirely is why an exotic format "is not supported" — the format is fine, your GDAL build just was not compiled with it.

OGR: the vector half, and why the name persists

GDAL began in 1998 as a raster library. OGR ("OpenGIS Simple Features Reference Implementation") arrived as a separate vector library, and the two merged in GDAL 2.0 in 2015. The names survive in the tooling:

gdalinfo data/raw/dem.tif          # raster metadata
ogrinfo -so -al data/raw/parcels.gpkg   # vector metadata

gdal_translate -of COG in.tif out.tif   # raster conversion
ogr2ogr -f GPKG out.gpkg in.shp         # vector conversion

In Python, the split shows up as from osgeo import gdal, ogr, osr — three modules of one library. Modern code rarely uses them directly, because rasterio and pyogrio are far more pleasant, but they are the same functionality.

Drivers: where the format limits live

Every quirk you have met is a driver behaviour:

import pyogrio

for name in ["ESRI Shapefile", "GPKG", "GeoJSON"]:
    print(f"{name:16} {pyogrio.list_drivers().get(name)}")

The shapefile driver truncates field names to ten bytes because dBase does. The GeoJSON driver writes WGS84 because RFC 7946 says so. The GPKG driver supports multiple layers because SQLite does. None of this is GeoPandas — GeoPandas simply passes your data to a driver and reports what happens.

Driver behaviour is configurable through open and creation options:

import geopandas as gpd

# creation options are passed through to the driver
gdf.to_file("out/parcels.geojson", driver="GeoJSON",
            COORDINATE_PRECISION=6, RFC7946="YES")

gdf.to_file("out/parcels.shp", driver="ESRI Shapefile", encoding="utf-8")

The virtual filesystem: reading things that are not files

One of GDAL's most useful and least known features is /vsi*/, a set of virtual filesystem handlers that make archives, memory and cloud storage look like paths.

import geopandas as gpd

# inside a zip, without unpacking
gpd.read_file("/vsizip/data/raw/parcels.zip/parcels.shp")

# straight from object storage
gpd.read_file("/vsis3/my-bucket/parcels/parcels.gpkg")
gpd.read_file("/vsicurl/https://example.org/data/buildings.fgb")

# gzip, tar and nesting all work
gpd.read_file("/vsigzip/data/raw/roads.geojson.gz")
gpd.read_file("/vsizip//vsicurl/https://example.org/delivery.zip/parcels.shp")

Combined with a format that supports partial reads, this is how a bbox query against a remote FlatGeobuf transfers only the bytes it needs.

Configuration through environment variables

GDAL is configured by environment variables far more than by API calls, which is why containers and cron jobs behave differently from laptops:

import os

os.environ["GDAL_CACHEMAX"] = "512"                    # MB of block cache
os.environ["GDAL_NUM_THREADS"] = "ALL_CPUS"            # threaded compression, warping
os.environ["GDAL_DISABLE_READDIR_ON_OPEN"] = "EMPTY_DIR"   # essential for /vsis3 speed
os.environ["CPL_DEBUG"] = "ON"                         # verbose driver logging
os.environ["OGR_GEOMETRY_ACCEPT_UNCLOSED_RING"] = "NO"

CPL_DEBUG=ON is the diagnostic to remember: CPL is GDAL's portability layer, and turning it on prints exactly which driver was tried, which file was opened and why something failed.

PROJ: the other library underneath

Coordinate transformations are not GDAL's own work; they are PROJ's. That separation explains a whole family of errors:

import pyproj

print("PROJ version:", pyproj.proj_version_str)
print("PROJ data dir:", pyproj.datadir.get_data_dir())
print(pyproj.CRS.from_epsg(27700).name)
ERROR 1: PROJ: proj_create_from_database: Cannot find proj.db

That message means PROJ cannot find its coordinate-operation database — usually because PROJ_LIB/PROJ_DATA points somewhere wrong, or because two PROJ installations are fighting. It is never a problem with your data.

Why installations break 

import rasterio, pyogrio, fiona
print("rasterio's GDAL:", rasterio.__gdal_version__)
print("pyogrio's GDAL :", pyogrio.__gdal_version_string__)
print("fiona's GDAL   :", fiona.__gdal_version__)

Three numbers that should match. They diverge because there are two ways to get GDAL into a Python environment:

  • wheels — pip install rasterio downloads a wheel with its own GDAL bundled inside
  • system library — conda, apt or a Docker base image provides one shared GDAL

Mixing them puts two GDAL builds in one process, which produces symbol errors, PROJ database errors and occasional crashes. The fix is to pick one strategy: all wheels, or all conda/system packages.

Code examples

Example 1: an environment report worth keeping 

"""gdal_env.py — print everything needed to diagnose a GIS environment."""
import sys, os

def report():
    print("python        :", sys.version.split()[0], "|", sys.executable)

    try:
        from osgeo import gdal
        print("gdal (osgeo)  :", gdal.__version__, "|", gdal.__file__)
    except ImportError:
        print("gdal (osgeo)  : not installed (fine — wrappers do not need it)")

    for module, attr in [("rasterio", "__gdal_version__"),
                         ("pyogrio", "__gdal_version_string__"),
                         ("fiona", "__gdal_version__")]:
        try:
            mod = __import__(module)
            print(f"{module:14}: GDAL {getattr(mod, attr)} | {mod.__file__}")
        except ImportError:
            print(f"{module:14}: not installed")

    import pyproj
    print("proj          :", pyproj.proj_version_str, "|", pyproj.datadir.get_data_dir())

    for var in ["GDAL_DATA", "PROJ_LIB", "PROJ_DATA", "GDAL_CACHEMAX", "GDAL_NUM_THREADS"]:
        print(f"  ${var:16} = {os.environ.get(var, '<unset>')}")

    import pyogrio
    drivers = pyogrio.list_drivers()
    print(f"vector drivers: {len(drivers)}")
    for key in ["ESRI Shapefile", "GPKG", "GeoJSON", "FlatGeobuf", "Parquet", "PostgreSQL"]:
        print(f"  {key:16} {drivers.get(key, 'MISSING')}")

report()

Run this first whenever a GIS environment misbehaves — it answers most questions before you start guessing.

Example 2: use the virtual filesystem instead of downloading 

import geopandas as gpd
import os

os.environ["GDAL_DISABLE_READDIR_ON_OPEN"] = "EMPTY_DIR"     # avoid listing the whole bucket
os.environ["VSI_CACHE"] = "TRUE"

# read a bbox from a remote FlatGeobuf: only the needed byte ranges are fetched
url = "/vsicurl/https://example.org/data/buildings.fgb"
subset = gpd.read_file(url, bbox=(-3.22, 55.93, -3.15, 55.97))
print(f"{len(subset):,} buildings without downloading the file")

# and a shapefile inside a zip inside an HTTP server
nested = gpd.read_file("/vsizip//vsicurl/https://example.org/delivery.zip/parcels.shp")

Example 3: driving ogr2ogr from Python for bulk work 

import subprocess
from pathlib import Path

def ogr2ogr(src, dest, *, layer=None, where=None, t_srs=None, fmt="GPKG", extra=()):
    cmd = ["ogr2ogr", "-f", fmt, str(dest), str(src)]
    if layer:  cmd += [layer]
    if where:  cmd += ["-where", where]
    if t_srs:  cmd += ["-t_srs", t_srs]
    cmd += ["-progress", *extra]
    proc = subprocess.run(cmd, capture_output=True, text=True)
    if proc.returncode != 0:
        raise RuntimeError(f"ogr2ogr failed: {proc.stderr.strip()[-500:]}")
    return Path(dest)

ogr2ogr("data/raw/parcels.shp", "data/out/parcels.gpkg",
        where="class = 'residential'", t_srs="EPSG:27700")

For a straight format conversion or a bulk load of millions of features, ogr2ogr is usually faster than reading into Python and writing back out — it never builds the objects.

Example 4: capture GDAL's own errors properly 

import logging
from osgeo import gdal

gdal.UseExceptions()                     # turn silent error codes into Python exceptions

logging.basicConfig(level=logging.INFO)
log = logging.getLogger("gdal")

def gdal_error_handler(err_class, err_num, message):
    level = {gdal.CE_Warning: log.warning, gdal.CE_Failure: log.error,
             gdal.CE_Fatal: log.critical}.get(err_class, log.info)
    level("GDAL %s: %s", err_num, message.strip())

gdal.PushErrorHandler(gdal_error_handler)

try:
    ds = gdal.OpenEx("data/raw/missing.gpkg", gdal.OF_VECTOR)
except RuntimeError as exc:
    print("open failed:", exc)
finally:
    gdal.PopErrorHandler()

gdal.UseExceptions() is worth setting in any script that touches the raw bindings: without it, GDAL signals failure by returning None and setting an error code, which is very easy to ignore.

Explanation

GDAL exists because the alternative is unthinkable. Without it, every GIS application would need its own reader for shapefile, GeoPackage, GeoJSON, KML, GML, MapInfo TAB, File Geodatabase, PostGIS, GeoTIFF, JPEG2000, NetCDF, HDF and a hundred more — each with its own bugs and its own gaps. Frank Warmerdam started it in 1998 for exactly this reason, and the result is that essentially the entire industry, open source and proprietary alike, reads data through one library.

Triage table mapping common GDAL-related errors to their real causes.
Six errors that name GDAL, PROJ or CPL — and what each one is actually telling you.

The abstraction has a shape worth understanding. A driver knows one format. A dataset is an open file or connection. A vector dataset has layers, each with a feature definition (the field schema and geometry type) and features. A raster dataset has bands, a geotransform and a CRS. Every Python wrapper you use is presenting some part of that model — pyogrio.read_info() is a feature-definition query; rasterio's profile is the raster dataset's metadata.

Because the model is uniform but the formats are not, GDAL has to decide what to do when a format cannot express something. Those decisions are the driver behaviours that surface as quirks: truncated field names, dropped time components, reprojected GeoJSON, a boolean stored as an integer. When something surprising happens on write, the question to ask is "what did this driver do?", and the answer is in GDAL's driver documentation rather than in GeoPandas'.

PROJ is the second pillar, and keeping the two straight prevents a lot of confusion. GDAL reads and writes; PROJ transforms coordinates and holds the EPSG database. A CRS error mentioning proj.db is a PROJ installation problem. A driver error mentioning CPLE_OpenFailed is a GDAL one. Both can be triggered by the same broken environment, but they have different fixes.

Finally, this explains the install advice that gets repeated everywhere. pip install geopandas now works well because the wheels bundle GDAL, PROJ and GEOS. It goes wrong when a system GDAL is also present — two copies of the same library in one process, disagreeing about versions and data paths. Choosing one source for the whole stack, and checking the versions agree, is the single most effective preventative measure available.

Edge cases or notes

  • gdal.UseExceptions() will be the default: GDAL 4 makes exceptions mandatory. Set it now to avoid silent None returns.
  • Driver availability varies by build: conda-forge, apt, wheels and the QGIS bundle all compile different driver sets. Check with pyogrio.list_drivers() rather than assuming.
  • GDAL_DATA and PROJ_LIB are usually set for you: Setting them by hand is a common way to break an otherwise working install.
  • /vsicurl/ needs a range-capable server: Without HTTP range support it downloads the whole file.
  • ogr2ogr and Python may use different GDALs: The CLI comes from the system; the Python wrapper may bundle its own. Compare versions when results differ.
  • The bindings are not the wrappers: osgeo.ogr is the raw SWIG binding; pyogrio/fiona are friendlier. Mixing them in one script is legal but confusing.
  • GDAL is thread-safe with care: Dataset objects are not shareable between threads. Open per thread, or use processes.

FAQ

What is the difference between GDAL and OGR?

Historically GDAL handled rasters and OGR handled vectors. They merged in GDAL 2.0 (2015); "OGR" now survives only in tool names like ogr2ogr and in the osgeo.ogr module.

Do I need to install GDAL separately for GeoPandas?

Not any more. Modern wheels for pyogrio, fiona, rasterio and shapely bundle the native libraries. Problems arise when a system GDAL is also present and the two disagree.

Why does an error mention CPLE or CPL?

CPL is GDAL's portability layer, so CPLE_* errors come from GDAL itself — file opening, driver selection, HTTP access. Set CPL_DEBUG=ON to see the full driver trace.

What is /vsizip/ and /vsis3/?

GDAL's virtual filesystem handlers. They let you open files inside archives, in memory, or in cloud storage using ordinary path syntax, with no unpacking or downloading step.

Why do two libraries report different GDAL versions?

Because each bundled its own copy, or one is using the system library. Two GDALs in one process cause symbol and PROJ database errors — standardise on wheels or on conda, not both.

Is ogr2ogr faster than doing it in Python?

For pure format conversion or bulk loading, usually yes: it streams features through GDAL without ever constructing Python objects. Use Python when you need per-feature logic.

What does PROJ do that GDAL does not?

PROJ owns coordinate reference systems and transformations, including the EPSG database in proj.db. GDAL calls into it. That is why CRS errors and format errors have different causes and different fixes.