⚙️ How-To Cloud-native formats Raster & rasterio Cloud-Native GIS · step 3 of 14 Updated

How to Read a COG from a URL Without Downloading the Whole File

Problem statement

A Cloud-Optimised GeoTIFF can be read in place: rasterio.open("https://...") and a windowed read fetches only the tiles it needs. That works out of the box, and out of the box it is slower than it should be.

Measured against a byte-counting server, four scattered 512 × 512 window reads from one COG:

default settings                 10 requests   4.610 MB
GDAL_DISABLE_READDIR_ON_OPEN      6 requests   4.155 MB
+ VSI_CACHE                        7 requests   4.171 MB

Four of the ten requests were GDAL probing for sidecar files that do not exist. On a local network that is invisible; against an object store 100 ms away it is 400 ms added to every file you open.

Quick answer 

import rasterio
from rasterio.windows import Window

SETTINGS = {
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
    "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.TIF,.tiff",
    "VSI_CACHE": "TRUE",
    "VSI_CACHE_SIZE": "50000000",
    "GDAL_HTTP_MERGE_CONSECUTIVE_RANGES": "YES",
}

with rasterio.Env(**SETTINGS):
    with rasterio.open("https://example.com/scene.tif") as ds:
        window = Window(1800, 1800, 512, 512)
        data = ds.read(1, window=window)

Set the environment once, at the top of the process, rather than per call. Each rasterio.Env block re-applies the configuration and clears some caches.

A remote COG read as a header request followed by tile range requests, with sidecar probes removed by GDAL_DISABLE_READDIR_ON_OPEN.
Two requests to open, then one range per contiguous run of tiles. The probes are pure overhead.

Step-by-step solution

1. Turn off directory listing and sidecar probing

GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR stops GDAL listing the "directory" containing the file, which on an object store is an expensive prefix query.

CPL_VSIL_CURL_ALLOWED_EXTENSIONS restricts which sidecar files GDAL will look for — .aux.xml, .tfw, .ovr, .msk. Each missing one is a round trip.

Together these removed four of ten requests in the benchmark above.

2. Read windows, not whole bands 

window = rasterio.windows.from_bounds(*aoi_bounds, transform=ds.transform)
data = ds.read(1, window=window.round_offsets().round_lengths())

The saving is unbounded in the file size: a 512 × 512 window cost 1.59 MB from a 33 MB file, and would cost the same 1.59 MB from a 209 MB file.

3. Align windows to the tile grid

Tiles are compressed independently, so the smallest readable unit is a whole tile. Measured on a COG with 512-pixel tiles:

      1 px    4 requests     470.8 kB
     64 px    4 requests     470.8 kB
    256 px    4 requests     470.8 kB
    512 px    5 requests   1,585.3 kB

A single pixel costs a whole tile. If you need scattered values, read the tile block containing them once rather than issuing a read per point.

4. Use out_shape for downsampled reads 

thumbnail = ds.read(1, out_shape=(ds.height // 16, ds.width // 16),
                    resampling=Resampling.average)

GDAL picks the appropriate overview level automatically. Measured: 1 request and 0.115 MB, against 24.672 MB from the same data without overviews.

Never read at full resolution and downsample in NumPy. That is the code path that transfers the whole file.

5. Reuse the dataset handle

Opening a remote dataset costs two requests and the header parse. In a loop over many windows, open once and read many times — the tile cache then works across reads.

A window straddling four tiles reading all four, against a tile-aligned window reading one.
An unaligned 512-pixel window can touch four tiles. The same window aligned touches one.

Code examples

Example 1 — a session-level configuration 

import os
import rasterio

REMOTE_SETTINGS = {
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
    "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.TIF,.tiff,.vrt",
    "VSI_CACHE": "TRUE",
    "VSI_CACHE_SIZE": str(64 * 1024 * 1024),
    "GDAL_CACHEMAX": 512,                       # MB of decoded block cache
    "GDAL_HTTP_MERGE_CONSECUTIVE_RANGES": "YES",
    "GDAL_HTTP_MULTIPLEX": "YES",
    "GDAL_HTTP_MAX_RETRY": 3,
    "GDAL_HTTP_RETRY_DELAY": 1,
    "GDAL_BAND_BLOCK_CACHE": "HASHSET",
}


def configure_remote_reads():
    """Apply once per process, before any dataset is opened."""
    for key, value in REMOTE_SETTINGS.items():
        os.environ.setdefault(key, str(value))
    print(f"  configured {len(REMOTE_SETTINGS)} GDAL options for remote reads")

Setting these as process environment variables rather than inside rasterio.Env blocks means they apply to every read including those inside libraries you do not control — rioxarray, stackstac, rasterstats.

The retry settings matter in production: object stores return transient 500s and 503s, and without retries a batch job fails on one bad response out of ten thousand.

Example 2 — reading many windows efficiently 

import numpy as np
import rasterio
from rasterio.windows import Window


def read_windows(url, windows, band=1):
    """One open, many reads, so the tile cache is shared."""
    results = []
    with rasterio.open(url) as ds:
        block_h, block_w = ds.block_shapes[band - 1]
        print(f"  tiles are {block_w} x {block_h}")

        # order windows by tile so nearby reads hit the cache
        def tile_key(w):
            return (int(w.row_off // block_h), int(w.col_off // block_w))

        for window in sorted(windows, key=tile_key):
            results.append(ds.read(band, window=window))

    print(f"  read {len(results)} windows from one open handle")
    return results


def align_to_tiles(window, block_w, block_h):
    """Expand a window outward to whole tiles."""
    col0 = int(window.col_off // block_w) * block_w
    row0 = int(window.row_off // block_h) * block_h
    col1 = int(np.ceil((window.col_off + window.width) / block_w)) * block_w
    row1 = int(np.ceil((window.row_off + window.height) / block_h)) * block_h
    return Window(col0, row0, col1 - col0, row1 - row0)

Sorting windows by tile is the cheapest optimisation available. Random-order reads evict the cache constantly; tile-ordered reads hit it repeatedly.

Example 3 — sampling points without a request per point 

import numpy as np
import rasterio
from rasterio.windows import Window


def sample_points(url, xs, ys, band=1, pad=0):
    """Read one block covering all the points, then index it."""
    with rasterio.open(url) as ds:
        rows, cols = rasterio.transform.rowcol(ds.transform, xs, ys)
        rows, cols = np.asarray(rows), np.asarray(cols)

        row0, row1 = max(rows.min() - pad, 0), min(rows.max() + pad + 1, ds.height)
        col0, col1 = max(cols.min() - pad, 0), min(cols.max() + pad + 1, ds.width)
        window = Window(col0, row0, col1 - col0, row1 - row0)

        span = (row1 - row0) * (col1 - col0)
        print(f"  {len(xs)} points span {col1 - col0} x {row1 - row0} px "
              f"({span / 1e6:.1f} Mpx) — reading as one block")
        if span > 50e6:
            print("  ! the points are spread too widely for one block; "
                  "cluster them and read per cluster")

        block = ds.read(band, window=window)
    return block[rows - row0, cols - col0]

ds.sample() is convenient and issues a read per point. For a hundred points inside one tile that is a hundred fetches of the same 470 kB tile — the cache absorbs most of it, but the block read is unambiguous and faster.

Where points are spread across the raster, cluster them spatially and read one block per cluster.

Explanation

Why the sidecar probes cost so much

When GDAL opens a dataset it looks for auxiliary files: .aux.xml for statistics, .ovr for external overviews, .msk for masks, world files for georeferencing.

On a local filesystem these are cheap stat calls. Over HTTP each is a request, and a 404 costs a full round trip.

GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR tells GDAL to assume the directory is empty, so it skips them. CPL_VSIL_CURL_ALLOWED_EXTENSIONS narrows which it will consider even when listing is allowed.

Measured: four requests removed from a ten-request read, before any data was transferred.

Why the tile is the unit

COG tiles are compressed independently. A client cannot decompress half a tile, so the minimum transfer for any pixel is that pixel's whole compressed tile.

That is why the benchmark shows 1 pixel, 64 × 64 and 256 × 256 all costing 470.8 kB — all three touch exactly one 512 × 512 tile plus the header.

The practical rule: make your access pattern tile-shaped. Sampling a raster at scattered points is close to the worst case; reading blocks is close to the best.

Why caching helps less than you expect

VSI_CACHE caches raw byte ranges; GDAL_CACHEMAX caches decoded blocks. Both help when the same tiles are read repeatedly.

In the benchmark, adding VSI_CACHE changed 6 requests to 7 and the bytes barely moved — because each of the four windows touched different tiles, so there was nothing to reuse.

Caching pays when access is repetitive or clustered: a tile server rendering adjacent tiles, a time series reading the same window from many files, a zoom-out followed by a zoom-in. Ordering reads to create that repetition is worth more than increasing the cache size.

Why to open once

Opening a remote dataset costs two requests and a header parse. Inside a loop that is two requests per iteration for nothing.

Worse, a fresh open resets the block cache, so windows that would have hit it miss instead. Keeping the handle open across a loop is often a larger saving than any environment setting.

Four scattered window reads taking ten requests by default and six with GDAL_DISABLE_READDIR_ON_OPEN.
On a high-latency link this one setting matters more than everything else combined.

Edge cases or notes

  • Set the environment once per process, not per read.
  • GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR is the single highest-value setting.
  • Use out_shape for downsampled reads, never full-resolution plus NumPy.
  • A 1-pixel read costs a whole tile — 470 kB here.
  • Sort windows by tile so the cache is used.
  • Open once, read many. A fresh open costs two requests and clears the cache.
  • Set retries. Object stores return transient errors, and one 503 should not fail a batch.
  • Requester-pays buckets need credentials and AWS_REQUEST_PAYER=requester.

FAQ

How do I read a COG from a URL in Python?

rasterio.open("https://...") and read a window. Set GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR first — it removed four of ten requests in a benchmark here.

How much data does a windowed read transfer?

Only the tiles the window touches. A 512 × 512 window cost 1.59 MB from a 33 MB file, and would cost the same from a much larger one.

Why does reading a single pixel take so long?

Because tiles are compressed independently, so the smallest transfer is one whole tile — 470 kB on a 512-tile COG.

How do I read a thumbnail cheaply?

ds.read(1, out_shape=(h // 16, w // 16), resampling=Resampling.average). GDAL selects the right overview: 1 request and 0.115 MB, against 24.7 MB without overviews.

Should I set VSI_CACHE?

Yes, but expect little from it unless your reads are clustered or repetitive. In a four-scattered-window benchmark it changed nothing.

Why open the file once?

A fresh open costs two requests and clears the block cache. In a loop over windows, opening once is often the largest single saving.

What about authentication?

For S3, set credentials in the environment and AWS_REQUEST_PAYER=requester for requester-pays buckets. Public HTTPS needs nothing.