Cloud-Native Geospatial Explained: Why the Format Became the API
Problem statement
The traditional way to use a remote dataset is to download it. That stops working when the dataset is a petabyte, and it was always wasteful when you needed a hundredth of it.
Cloud-native geospatial formats invert the arrangement: the file stays where it is, and the client reads only the bytes it needs using HTTP range requests. The format itself becomes the API — no server, no database, no query language.
The saving is not marginal. Reading a full-resolution 512 × 512 window from a 33 MB Cloud-Optimised GeoTIFF over HTTP transferred 1.59 MB in 5 requests. Reading a whole-scene thumbnail from the same file transferred 0.115 MB in 1 request, against 24.7 MB — the entire file — from a tiled GeoTIFF without overviews.
Quick answer
Four properties make a format cloud-native:
1. internally tiled or chunked so a subset is a contiguous byte range
2. an index in the header so the client knows which range to ask for
3. overviews or multiscale levels so a low-resolution read is cheap
4. served over HTTP with ranges so the client can ask for part of it
import rasterio
with rasterio.Env(GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR",
CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".tif"):
with rasterio.open("https://example.com/scene.tif") as ds:
window = ds.read(1, window=Window(1800, 1800, 512, 512))No download step. GDAL reads the header, works out which internal tiles the window touches, and fetches those byte ranges.
Step-by-step solution
1. Internal tiling: so a subset is contiguous
A striped GeoTIFF stores the image row by row. A 512 × 512 window touches 512 separate strips scattered through the file.
A tiled GeoTIFF stores it in blocks — typically 256 or 512 pixels square — so the same window is a handful of contiguous ranges. Measured on the same 4096 × 4096 raster:
one 512 x 512 window
COG (512 tiles, overviews) 5 requests 1.585 MB
tiled, no overviews 3 requests 1.557 MB
striped 2 requests 3.244 MB
The striped file transferred twice the bytes for the same pixels, because each strip spans the full 4096-pixel width and only 512 of those pixels were wanted.
2. An index: so the client knows what to ask for
Tiling alone is not enough — the client must know where each tile lives. In a GeoTIFF that is the tile offset array in the header, and "cloud-optimised" partly means arranging the file so the header is at the front and readable in one request.
Measured, opening a COG over HTTP and reading nothing cost 2 requests and 28 kB.
3. Overviews: so a low-resolution read is cheap
This is the property with the largest effect, and the one most often missing:
whole scene at 1/16 resolution
COG (with overviews) 1 request 0.115 MB
tiled, no overviews 3 requests 24.672 MB
Without overviews the client must read every full-resolution pixel and average them down. That is the entire file — a 215-fold difference in bytes for an identical output.
Any application that zooms — a viewer, a tile server, a thumbnail generator — lives or dies on this.
4. HTTP range requests: so partial reads are possible
The server must support Range headers and return 206 Partial Content. Almost all object stores do; some CDNs and application servers do not.
Where ranges are unsupported, every read fetches the whole file and the other three properties buy nothing.
5. Know what the format cannot do
Cloud-native formats are excellent at "give me this rectangle at this resolution". They are not databases. They cannot join, cannot index by attribute, and cannot answer "where is the value above 3,000" without reading everything.
The right architecture is usually a catalogue for finding data — STAC — and cloud-native files for reading it.
Code examples
Example 1 — measuring what a read actually costs
import time
import rasterio
from rasterio.windows import Window
def measure_read(url, window=None, out_shape=None, env=None):
"""Time a read and report what GDAL was configured to do."""
settings = env or dict(GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR",
CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".tif")
start = time.time()
with rasterio.Env(**settings):
with rasterio.open(url) as ds:
opened = time.time() - start
data = ds.read(1, window=window, out_shape=out_shape)
total = time.time() - start
print(f" {ds.width}x{ds.height}, blocks {ds.block_shapes[0]}, "
f"overviews {ds.overviews(1)}")
print(f" open {opened * 1000:.0f} ms, total {total * 1000:.0f} ms, "
f"returned {data.shape}")
if not ds.overviews(1):
print(" ! no overviews — any downsampled read costs the "
"full resolution")
return dataThe overview warning is the single most useful line. A file without overviews behaves acceptably for full-resolution windows and catastrophically for anything zoomed out, and nothing else in the output reveals it.
Example 2 — writing a file that is actually cloud-optimised
import rasterio
from rasterio.enums import Resampling
def write_cog(src_path, dst_path, blocksize=512, overview_levels=(2, 4, 8, 16),
compress="deflate", predictor=2):
"""Tiled, compressed, with overviews and the header at the front."""
with rasterio.open(src_path) as src:
profile = src.profile | {
"driver": "GTiff", "tiled": True,
"blockxsize": blocksize, "blockysize": blocksize,
"compress": compress, "predictor": predictor,
"BIGTIFF": "IF_SAFER",
}
with rasterio.open(dst_path, "w", **profile) as dst:
for i in range(1, src.count + 1):
dst.write(src.read(i), i)
dst.build_overviews(list(overview_levels), Resampling.average)
dst.update_tags(ns="rio_overview", resampling="average")
with rasterio.open(dst_path) as check:
print(f" {dst_path}: blocks {check.block_shapes[0]}, "
f"overviews {check.overviews(1)}")
if not check.overviews(1):
raise RuntimeError("overviews were not written")Three details decide whether the result is genuinely cloud-optimised: tiled=True, real overviews, and a layout with the header first. GDAL's COG driver handles the last one; writing with the plain GTiff driver and adding overviews afterwards usually produces an acceptable file, and validating it is worth the extra check.
Example 3 — GDAL settings that matter for remote reads
import rasterio
from rasterio.windows import Window
SETTINGS = {
# do not list the whole "directory" on the object store
"GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
# do not probe for sidecar files with other extensions
"CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.TIF,.tiff",
# keep decoded blocks around between reads
"VSI_CACHE": "TRUE",
"VSI_CACHE_SIZE": "50000000",
# merge nearby ranges into one request
"GDAL_HTTP_MERGE_CONSECUTIVE_RANGES": "YES",
}
def tuned_read(url, windows):
with rasterio.Env(**SETTINGS):
with rasterio.open(url) as ds:
return [ds.read(1, window=w) for w in windows]Measured on four scattered windows from one COG:
no tuning 10 requests 4.610 MB
GDAL_DISABLE_READDIR_ON_OPEN 6 requests 4.155 MB
+ VSI_CACHE 7 requests 4.171 MB
GDAL_DISABLE_READDIR_ON_OPEN removed four requests — those were GDAL probing for sidecar files that do not exist. On a high-latency connection each probe is a full round trip, so this one setting often matters more than everything else combined.
Explanation
Why range requests change the architecture
Before range requests, remote data meant a service: a WMS, a WCS, a tile server, a database. The server held the data, understood the format and answered queries.
With range requests, the client understands the format and the server only has to serve bytes. That removes an entire tier: no application server to run, scale, secure or pay for, and static object storage is the cheapest hosting there is.
The trade is that the client does more work and must speak the format. It also means every optimisation has to live in the file, which is exactly what "cloud-optimised" means.
Why overviews dominate
The measured difference is 0.115 MB against 24.672 MB for the same thumbnail — 215×.
The reason is structural. Without overviews, producing a downsampled image requires reading every source pixel, because averaging needs the values. With overviews, the averaging was done once at write time and stored, so the client reads a small pre-computed array.
Every zoomable application does far more low-resolution reads than full-resolution ones — a map viewer at zoom 10 is reading overviews almost exclusively. A file without them makes that pattern impossible.
Why the same idea appears in every format
The pattern generalises, and each format applies it to its own data model:
- COG — tiles and overviews for rasters.
- Zarr — chunks and a JSON metadata file for n-dimensional arrays; multiscale by convention.
- GeoParquet — row groups with per-column statistics for tables, so a reader can skip groups.
- COPC — an octree inside a LAZ file for point clouds, giving both spatial selection and level of detail.
- PMTiles — a directory of tiles inside a single archive with a header index.
All five are the same three ideas: chunk the data, index the chunks, and provide coarser levels.
Why the tuning settings matter more than the bandwidth
Four scattered window reads cost 10 requests with default settings and 6 with GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR.
The four removed requests were GDAL checking for sidecar files — a .tif.aux.xml, a .tfw, an .ovr — that do not exist. Each is a full HTTP round trip.
On a local network that costs microseconds. Against an object store 100 ms away, four round trips is 400 ms added to every file opened, which dominates any bandwidth consideration for small reads.
Edge cases or notes
- Tiled without overviews is half a COG. Downsampled reads still cost the whole file.
- Set
GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIRfor remote reads — it removed 4 of 10 requests here. - The server must support
Range. Some CDNs and app servers do not. - Compression trades bytes for CPU. Deflate with a predictor is a good default for continuous data.
- A 1 × 1 pixel read still costs a whole tile — 471 kB on a 512-tile COG.
- Cloud-native formats are not databases. Pair them with a catalogue such as STAC.
- Requester-pays buckets need credentials even for public data.
- Validate after writing. A file can be tiled, compressed and still not have overviews.
Internal links
- Cloud-optimised GeoTIFF explained: tiles, overviews, range requests — the raster case in detail
- How to read a COG from a URL without downloading the whole file — the practical guide
- How to write a cloud-optimised GeoTIFF in Python — producing one
- Chunked arrays and Zarr explained — the same idea for n-dimensional data
- Partitioning explained: how GeoParquet gets fast on cloud storage — the tabular case
- Reading a COG from a URL is slow or downloads everything — when it goes wrong
- STAC explained: how satellite imagery catalogues work — the catalogue half of the architecture
- LAS, LAZ and COPC explained — the point cloud case
FAQ
What does cloud-native geospatial mean?
A format arranged so a client can read part of it over HTTP with range requests: internally tiled or chunked, with an index in the header and pre-computed coarser levels.
Do I still need a server?
Usually not. Static object storage that supports range requests is enough, which removes an entire tier of infrastructure.
How much does a COG actually save?
Measured on a 33 MB file: a 512 × 512 window cost 1.59 MB, and a whole-scene thumbnail cost 0.115 MB against 24.7 MB from the same data without overviews.
Is a tiled GeoTIFF a COG?
Not quite. Tiling helps full-resolution windows; without overviews, any downsampled read still costs the full resolution.
What GDAL settings should I use for remote reads?
GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR first — it removed four of ten requests in a four-window benchmark. Then CPL_VSIL_CURL_ALLOWED_EXTENSIONS and VSI_CACHE.
Can I query a cloud-native file by attribute?
Only in formats with column statistics, such as GeoParquet, and only coarsely. These are not databases; pair them with a catalogue for discovery.
Does this work for vector and point cloud data too?
Yes — GeoParquet for tables, PMTiles for vector tiles, COPC for point clouds. All apply the same chunk-index-multiscale pattern.