Cloud-native formats
COG, Zarr, GeoParquet and STAC: reading the part you need over the network instead of downloading the file.
Everything on Spatial Workflow about cloud-native formats, in the order that usually works: understand the idea, then run the task, then fix it when it goes wrong. 22 guides in total.
6 learning paths run through this topic — follow one if you would rather work through a route than pick a single guide.
🧭 Start with the concepts
The ideas the how-to guides below assume you already have.
- Cloud-Native Geospatial Explained: Why the Format Became the API cloud native geospatial explained
- Cloud-Optimised GeoTIFF Explained: Tiles, Overviews and Range Requests cloud optimised geotiff explained
- Lazy Loading Explained: Why xarray Reads Nothing Until You Ask lazy loading xarray explained
- Chunked Arrays and Zarr Explained zarr chunked arrays explained
- GRIB, NetCDF or Zarr: Choosing a Format for Gridded Data grib netcdf zarr difference
- Metadata standards compared: ISO 19115, STAC and Frictionless iso 19115 stac compared
⚙️ Then do the task
Step-by-step walkthroughs with code you can copy and adapt.
- How to Read a COG from a URL Without Downloading the Whole File read cog from url python
- How to Write a Cloud-Optimised GeoTIFF in Python write cloud optimised geotiff python
- How to Read Spatial Data from S3 and Other Object Storage read spatial data from s3 python
- How to Turn a STAC Search into an xarray Data Cube stac to xarray cube python
- How to Choose Chunk and Tile Sizes That Actually Help choose chunk and tile sizes
- How to Write Partitioned GeoParquet and Query It with Filters partitioned geoparquet python
- How to Scale a GeoPandas Job with dask-geopandas dask geopandas scale python
- How to Render Raster Tiles from a COG in Python raster tiles from cog
- How to Query Remote GeoParquet over HTTP with DuckDB duckdb remote geoparquet
- How to Query Hive-Partitioned GeoParquet in DuckDB duckdb hive partitioned parquet
- How to Serve Raster Tiles from a COG with TiTiler serve raster tiles titiler
- How to build a STAC item for your own raster build stac item python
🔧 When it breaks
The errors this topic produces, with the cause explained rather than just the command.
- Reading a COG from a URL Is Slow or Downloads Everything cog read is slow
- GDAL Cannot Open an S3 or HTTPS Path gdal cannot open s3 url
- dask-geopandas Uses More Memory or Time Than Plain GeoPandas dask geopandas slower than geopandas
- A STAC item is rejected by the validator stac item validation error
Learning paths through this topic
A curated route rather than a list — the guides in the order that builds on itself.
-
Cloud-Native GIS
Read the part you need, not the file it lives in.
Open path → -
Publishing Web Maps
Get the data into a browser without shipping the whole dataset.
Open path → -
DuckDB for Spatial Data
A spatial database with no server, reading the file you already have.
Open path → -
Serving Spatial Data
The dataset is finished. Now other people need it over HTTP.
Open path → -
Gridded & Climate Data
A raster with a time axis, a calendar and forty years in one file.
Open path → -
Metadata, Licensing & Provenance
A layer nobody can describe, cite or re-run is a layer nobody should use.
Open path →
Related topics
Subjects that keep coming up alongside this one, most overlap first.