🧭 Concept Cloud-native formats Cloud-Native GIS · step 1 Notebook By · Updated

Cloud-Native Geospatial Explained: Why the Format Became the API

On this page

Problem statement

The traditional way to use a remote dataset is to download it. That stops working when the dataset is a petabyte, and it was always wasteful when you needed a hundredth of it.

Cloud-native geospatial formats invert the arrangement: the file stays where it is, and the client reads only the bytes it needs using HTTP range requests. The format itself becomes the API — no server, no database, no query language.

The saving is not marginal. One Sentinel-2 scene's near-infrared band is a 225.2 MB Cloud-Optimised GeoTIFF on Amazon S3. Reading a full-resolution 512 × 512 window from it, in place, fetched 1.51 MB; reading the whole scene at a sixteenth of its resolution fetched 0.74 MB (Example 1).

Quick answer

Four properties make a format cloud-native:

1. internally tiled or chunked      so a subset is a contiguous byte range
2. an index in the header           so the client knows which range to ask for
3. overviews or multiscale levels   so a low-resolution read is cheap
4. served over HTTP with ranges     so the client can ask for part of it

import rasterio
from rasterio.windows import Window

# URL: a Sentinel-2 band on Amazon S3 (see "The data used on this page"); any http(s) COG works the same way
with rasterio.Env(GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR"):
    with rasterio.open(URL) as ds:
        window = ds.read(1, window=Window(4096, 4096, 512, 512))
print(window.shape, window.dtype)
(512, 512) uint16

No download step. GDAL reads the header, works out which internal tiles the window touches, and fetches those byte ranges.

Four properties of a cloud-native format: internal tiling, a header index, overviews, and HTTP range support. miss any one and the client reads the whole file1internally tiled or chunkeda subset is contiguous2an index in the headerwhich bytes to ask for3overviews or multiscale levelscheap zoomed-out reads4HTTP range requests206 Partial ContentThe format becomes the API: no server, no database, no query language spatialworkflow.io
Miss any one and the client falls back to reading the whole file.

Tiling, an index, overviews and range requests

1. Internal tiling: so a subset is contiguous

A striped GeoTIFF stores the image row by row, so a 512 × 512 window touches 512 strips, each spanning the full width of the image; the slow-read guide measures what that costs.

A tiled GeoTIFF stores it in square blocks, so a window is a few contiguous ranges, and the tile becomes the unit of transfer. The Sentinel-2 band has 1,024-pixel tiles: a 512 × 512 window inside one tile fetched 1.51 MB, the same window placed across four tiles 6.17 MB, and a single pixel 1.51 MB — the whole tile it sits in (Example 1).

2. An index: so the client knows what to ask for

Tiling alone is not enough — the client must know where each tile lives. In a GeoTIFF that is the tile offset array in the header, and "cloud-optimised" partly means arranging the file so the header is at the front and readable in one request.

Opening the Sentinel-2 band and reading nothing cost one 16 kB range request once GDAL had asked S3 for the file's size (Example 1): the header, with the offsets of every tile, sits in the first 16 kB of a 225 MB file.

3. Overviews: so a low-resolution read is cheap

This is the property with the largest effect, and the one most often missing. Read at a sixteenth of its resolution, the whole 10,980 × 10,980 scene cost 0.74 MB, 0.33% of the file, because GDAL read the stored 1/16 overview instead of the full-resolution tiles.

Without overviews the client must read every full-resolution tile and average them down — most of the file. The slow-read guide measures that on a file written without them.

Any application that zooms — a viewer, a tile server, a thumbnail generator — lives or dies on this.

4. HTTP range requests: so partial reads are possible

The server must support Range headers and return 206 Partial Content. Almost all object stores do; some CDNs and application servers do not.

Where ranges are unsupported, every read fetches the whole file and the other three properties buy nothing.

5. Know what the format cannot do

Cloud-native formats are excellent at "give me this rectangle at this resolution". They are not databases. They cannot join, cannot index by attribute, and cannot answer "where is the value above 3,000" without reading everything.

The right architecture is usually a catalogue for finding data — STAC — and cloud-native files for reading it.

Reading a whole Sentinel-2 scene at 1/16 resolution from its overview cost 0.737 megabytes of a 225.2 megabyte file. whole scene at 1/16, bytes fetched from S3read from the overview0.737 MBthe whole file225.2 MB0120 MB240 MBThe overview turns the whole scene into a third of a percent of the file spatialworkflow.io
Overviews are the difference between a viewer that works and one that downloads the archive.

One Sentinel-2 band, read in place on Amazon S3

The data used on this page

The 10 m near-infrared band of Sentinel-2 scene S2A_30UVD_20250922_0_L2A, as Earth Search publishes it: a Cloud-Optimised GeoTIFF on Amazon S3, read where it lies. cost(fn) runs fn(url) on a URL GDAL has not cached — the same object with a new query string — and returns how many byte ranges GDAL fetched and how many bytes, counted from GDAL's own debug messages. Example 2 also writes a COG locally, from a 4096 × 4096 crop of the same band.

import itertools
import logging
import os
import re

import pystac_client
import rasterio
import requests
from rasterio.windows import Window

item = next(pystac_client.Client.open("https://earth-search.aws.element84.com/v1")
            .search(ids=["S2A_30UVD_20250922_0_L2A"], collections=["sentinel-2-l2a"]).items())
URL = item.assets["nir"].href                            # band 8, 10 m, on Amazon S3
SIZE = int(requests.head(URL, timeout=60).headers["Content-Length"])


class RangeLog(logging.Handler):
    """Collects the size of every byte range GDAL downloads, from its debug messages."""
    def __init__(self):
        super().__init__()
        self.sizes = []

    def emit(self, record):
        self.sizes += [int(b) - int(a) + 1
                       for a, b in re.findall(r"Downloading (\d+)-(\d+)", record.getMessage())]


LOG = RangeLog()
logging.getLogger("rasterio").addHandler(LOG)
logging.getLogger("rasterio").setLevel(logging.DEBUG)
_fresh = itertools.count()


def cost(fn):
    """Range requests and bytes GDAL fetched while fn(url) ran, starting from a cold cache."""
    LOG.sizes = []
    with rasterio.Env(CPL_DEBUG=True, GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR"):
        fn(f"{URL}?read={next(_fresh)}")                # a new URL, so nothing is cached
    return len(LOG.sizes), sum(LOG.sizes)


if not os.path.exists("nir.tif"):                        # a 4096 x 4096 crop, for Example 2
    with rasterio.open(URL) as src:
        win = Window(3000, 3000, 4096, 4096)
        profile = {"driver": "GTiff", "dtype": "uint16", "count": 1, "width": 4096, "height": 4096,
                   "crs": src.crs, "transform": src.window_transform(win)}
        with rasterio.open("nir.tif", "w", **profile) as dst:
            dst.write(src.read(1, window=win), 1)

with rasterio.open(URL) as ds:
    print(f"{URL.rsplit('/', 1)[-1]}: {SIZE / 1e6:.1f} MB, {ds.width} x {ds.height}, "
          f"blocks {ds.block_shapes[0]}, overviews {ds.overviews(1)}, {ds.compression.value.lower()}")
B08.tif: 225.2 MB, 10980 x 10980, blocks (1024, 1024), overviews [2, 4, 8, 16], deflate

Example 1 — what each read of the remote file costs

def open_only(url):
    with rasterio.open(url):
        pass


def read(**how):
    def run(url):
        with rasterio.open(url) as ds:
            ds.read(1, **how)
    return run


READS = {
    "open, read nothing": open_only,
    "512 x 512 window in one tile": read(window=Window(4096, 4096, 512, 512)),
    "512 x 512 window on 4 tiles": read(window=Window(4000, 4000, 512, 512)),
    "one pixel": read(window=Window(5000, 5000, 1, 1)),
    "whole scene at 1/16": read(out_shape=(686, 686)),
}
for label, fn in READS.items():
    n, sent = cost(fn)
    print(f"{label:30} {n} ranges {sent / 1e6:7.3f} MB  {sent / SIZE:6.2%} of the file")
open, read nothing             1 ranges   0.016 MB   0.01% of the file
512 x 512 window in one tile   2 ranges   1.507 MB   0.67% of the file
512 x 512 window on 4 tiles    3 ranges   6.173 MB   2.74% of the file
one pixel                      2 ranges   1.507 MB   0.67% of the file
whole scene at 1/16            2 ranges   0.737 MB   0.33% of the file

The tile, not the window, is what a read costs: one pixel fetched as much as a 512 × 512 window inside the same tile, and a window that crosses tile boundaries fetched all four tiles it touches. Each read here also re-reads the 16 kB header, because every URL is new to GDAL; a client that keeps a file open pays for the header once.

Range requests and bytes for opening a 225.2 MB Sentinel-2 COG on S3, reading windows inside and across tiles, one pixel, and the whole scene at 1/16. what each read of a 225.2 MB file cost on S3readrangesfetchedopen, read nothing10.016 MB512 × 512 window in one tile21.507 MB512 × 512 window on 4 tiles36.173 MBone pixel21.507 MBwhole scene at 1/1620.737 MBEvery read after the first re-fetches the 16 kB header, because each URL is new spatialworkflow.io
Example 1's measurements: the tile is the unit of transfer, and the overview makes the whole scene cheap.

Example 2 — writing a file that is actually cloud-optimised

import rasterio
from osgeo_utils.samples.validate_cloud_optimized_geotiff import validate
from rasterio.shutil import copy as rio_copy


def write_cog(src_path, dst_path, blocksize=512, overview_count=4, compress="DEFLATE"):
    """Tiled, compressed, with overviews and the header at the front — checked by GDAL's validator."""
    rio_copy(src_path, dst_path, driver="COG", BLOCKSIZE=blocksize, COMPRESS=compress,
             PREDICTOR="YES", OVERVIEW_RESAMPLING="AVERAGE", OVERVIEW_COUNT=overview_count,
             BIGTIFF="IF_SAFER")
    warnings, errors, _ = validate(dst_path, full_check=True)
    with rasterio.open(dst_path) as check:
        print(f"  {dst_path}: blocks {check.block_shapes[0]}, "
              f"overviews {check.overviews(1)}, validator errors {len(errors)}")
    if errors:
        raise RuntimeError(f"not a valid COG: {errors}")


write_cog("nir.tif", "cog_written.tif")
  cog_written.tif: blocks (512, 512), overviews [2, 4, 8, 16], validator errors 0

Three details decide whether the result is genuinely cloud-optimised: tiles, real overviews, and a layout with the header and overviews first. GDAL's COG driver handles all three. Writing with the plain GTiff driver and adding overviews afterwards gets the first two and not the third — GDAL's validator rejects such a file, and measured it costs an extra request on every window read.

Why the format replaced the server

Why range requests change the architecture

Before range requests, remote data meant a service: a WMS, a WCS, a tile server, a database. The server held the data, understood the format and answered queries.

With range requests, the client understands the format and the server only has to serve bytes. That removes an entire tier: no application server to run, scale, secure or pay for, and static object storage is the cheapest hosting there is.

The trade is that the client does more work and must speak the format. It also means every optimisation has to live in the file, which is exactly what "cloud-optimised" means.

Why overviews dominate

Here the whole scene at 1/16 cost 0.74 MB of a 225.2 MB file: 0.33% of it, for an image of the entire scene.

The reason is structural. Without overviews, producing a downsampled image requires reading every source pixel, because averaging needs the values. With overviews, the averaging was done once at write time and stored, so the client reads a small pre-computed array.

Every zoomable application does far more low-resolution reads than full-resolution ones — a map viewer at zoom 10 is reading overviews almost exclusively. A file without them makes that pattern impossible.

Why the same idea appears in every format

The pattern generalises, and each format applies it to its own data model:

  • COG — tiles and overviews for rasters.
  • Zarr — chunks and a JSON metadata file for n-dimensional arrays; multiscale by convention.
  • GeoParquet — row groups with per-column statistics for tables, so a reader can skip groups.
  • COPC — an octree inside a LAZ file for point clouds, giving both spatial selection and level of detail.
  • PMTiles — a directory of tiles inside a single archive with a header index.

All five are the same three ideas: chunk the data, index the chunks, and provide coarser levels.

The same chunk-index-multiscale pattern applied to rasters, arrays, tables, point clouds and tiles. one pattern, five data modelschunkindexmultiscaleCOGtilesheader offsets✓overviewsZarrchunksJSON metadata!by conventionGeoParquetrow groupscolumn statisticsnoneCOPCoctree nodeshierarchy✓octree depthPMTilestilesdirectory✓zoom levelsChunk the data, index the chunks, provide coarser levels spatialworkflow.io
Five formats, one idea. Only GeoParquet lacks a multiscale story, which is why it is paired with tiles.

Half a COG, Range support and what formats cannot do

  • Tiled without overviews is half a COG. Downsampled reads still cost the whole file.
  • Set GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR for remote reads, or GDAL probes the bucket for sidecar files that do not exist.
  • The server must support Range. Some CDNs and app servers do not.
  • Compression trades bytes for CPU. Deflate with a predictor is a good default for continuous data.
  • A 1 × 1 pixel read still costs a whole tile — 1.51 MB on the Sentinel-2 band's 1,024-pixel tiles (Example 1).
  • Cloud-native formats are not databases. Pair them with a catalogue such as STAC.
  • Requester-pays buckets need credentials even for public data.
  • Validate after writing. A file can be tiled, compressed and still not have overviews.

FAQ

What does cloud-native geospatial mean?

A format arranged so a client can read part of it over HTTP with range requests: internally tiled or chunked, with an index in the header and pre-computed coarser levels.

Do I still need a server?

Usually not. Static object storage that supports range requests is enough, which removes an entire tier of infrastructure.

How much does a COG actually save?

On a 225.2 MB Sentinel-2 band read in place on S3, a 512 × 512 window cost 1.51 MB and the whole scene at 1/16 resolution 0.74 MB.

Is a tiled GeoTIFF a COG?

Not quite. Tiling helps full-resolution windows; without overviews, any downsampled read still costs the full resolution.

Can I query a cloud-native file by attribute?

Only in formats with column statistics, such as GeoParquet, and only coarsely. These are not databases; pair them with a catalogue for discovery.

Does this work for vector and point cloud data too?

Yes — GeoParquet for tables, PMTiles for vector tiles, COPC for point clouds. All apply the same chunk-index-multiscale pattern.