⚙️ How-To Cloud-native formats Data sources & downloads Cloud-Native GIS · step 6 of 14 Updated

How to Read Spatial Data from S3 and Other Object Storage

Problem statement

Reading spatial data straight from object storage removes the download step, and adds a configuration surface that fails in ways local files never do: credentials, requester-pays, region endpoints, and per-request latency.

The failures are specific. A public dataset that needs credentials anyway:

RasterioIOError: AccessDenied: Anonymous users cannot invoke requests
against Requester Pays buckets.

That is the USGS Landsat Collection 2 archive — open data on a requester-pays bucket, where anonymous reads are refused and the reader pays the transfer. Sentinel-2 on AWS, by contrast, is openly readable with no account at all.

Quick answer 

import rasterio

# open, no credentials
with rasterio.Env(AWS_NO_SIGN_REQUEST="YES",
                  GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR"):
    with rasterio.open("s3://sentinel-cogs/.../B04.tif") as ds:
        data = ds.read(1, window=window)

# requester-pays: credentials AND the payer flag
with rasterio.Env(AWS_REQUEST_PAYER="requester",
                  GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR"):
    with rasterio.open("s3://usgs-landsat/.../SR_B4.TIF") as ds:
        data = ds.read(1, window=window)

AWS_NO_SIGN_REQUEST and AWS_REQUEST_PAYER are mutually exclusive: anonymous requests cannot be billed, so a requester-pays bucket rejects them regardless of the payer flag.

Three access modes for object storage: open anonymous, credentialed, and requester-pays where the reader is billed.
Open data is not the same as freely readable. Requester-pays needs an account before the first byte.

Step-by-step solution

1. Work out which access mode the bucket uses

mode settings example
public, anonymous AWS_NO_SIGN_REQUEST=YES Sentinel-2 COGs on AWS
public, requester-pays credentials + AWS_REQUEST_PAYER=requester USGS Landsat Collection 2
private credentials your own bucket

The middle row surprises people. The data is open, the licence is permissive, and you still need an AWS account because the storage owner does not pay for egress.

2. Use the right path form 

"s3://bucket/key"                                   # GDAL /vsis3, needs config
"/vsis3/bucket/key"                                 # the same, explicit
"https://bucket.s3.amazonaws.com/key"               # /vsicurl, public only

The https:// form goes through /vsicurl, which does no signing — it works for public objects and fails on anything requiring credentials. The s3:// form uses /vsis3, which handles signing, regions and requester-pays.

Other providers have their own prefixes: /vsigs/ for Google Cloud Storage, /vsiaz/ for Azure Blob Storage, /vsioss/ for Alibaba.

3. Set the region, or pay for a redirect 

"AWS_S3_ENDPOINT": "s3.us-west-2.amazonaws.com"

Requests to the wrong regional endpoint get a redirect, which is an extra round trip per request. On a windowed read making five requests, that is five extra round trips.

4. Set the same GDAL options as for any remote read 

"GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
"CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.TIF",
"VSI_CACHE": "TRUE",

Measured on a remote COG, GDAL_DISABLE_READDIR_ON_OPEN removed four of ten requests — all probes for sidecar files that did not exist. On object storage those probes are also ListObjects calls, which are billed separately from GetObject.

5. Set retries

Object stores return transient 500s and 503s under load. Without retries, one bad response in ten thousand fails a batch job.

"GDAL_HTTP_MAX_RETRY": 3,
"GDAL_HTTP_RETRY_DELAY": 1,
The s3 path form routing through vsis3 with signing and requester-pays support, against the https form through vsicurl with neither.
The https form cannot sign a request. On a private or requester-pays bucket it can only fail.

Code examples

Example 1 — configuration by bucket 

import os
import rasterio

PROFILES = {
    "sentinel-cogs": {
        "AWS_NO_SIGN_REQUEST": "YES",
        "AWS_S3_ENDPOINT": "s3.us-west-2.amazonaws.com",
    },
    "usgs-landsat": {
        "AWS_REQUEST_PAYER": "requester",
        "AWS_S3_ENDPOINT": "s3.us-west-2.amazonaws.com",
    },
}

COMMON = {
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
    "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.TIF,.tiff,.jp2",
    "VSI_CACHE": "TRUE",
    "VSI_CACHE_SIZE": str(64 * 1024 * 1024),
    "GDAL_HTTP_MAX_RETRY": 3,
    "GDAL_HTTP_RETRY_DELAY": 1,
}


def env_for(url):
    """GDAL settings appropriate to the bucket in this URL."""
    bucket = url.split("/")[2] if url.startswith("s3://") else ""
    settings = dict(COMMON)
    settings.update(PROFILES.get(bucket, {}))

    if settings.get("AWS_REQUEST_PAYER") and not os.environ.get("AWS_ACCESS_KEY_ID"):
        raise RuntimeError(
            f"{bucket} is requester-pays and no AWS credentials are set. "
            "Anonymous reads are refused regardless of the payer flag.")
    return rasterio.Env(**settings)


def read_remote(url, **kwargs):
    with env_for(url):
        with rasterio.open(url) as ds:
            return ds.read(**kwargs)

Raising before the read, with a message naming the bucket and the reason, turns a cryptic AccessDenied into an actionable one. On a requester-pays bucket the failure is otherwise indistinguishable from a permissions problem on your own data.

Example 2 — listing objects without downloading them 

import boto3


def list_prefix(bucket, prefix, requester_pays=False, suffix=".tif", limit=50):
    """Enumerate objects and their sizes before deciding what to read."""
    client = boto3.client("s3")
    kwargs = {"Bucket": bucket, "Prefix": prefix}
    if requester_pays:
        kwargs["RequestPayer"] = "requester"

    total_bytes, found = 0, []
    paginator = client.get_paginator("list_objects_v2")
    for page in paginator.paginate(**kwargs):
        for obj in page.get("Contents", []):
            if obj["Key"].endswith(suffix):
                found.append((obj["Key"], obj["Size"]))
                total_bytes += obj["Size"]
            if len(found) >= limit:
                break

    print(f"  {len(found)} objects, {total_bytes / 1e9:.2f} GB total")
    for key, size in found[:5]:
        print(f"    {size / 1e6:8.1f} MB  {key}")
    return found

Listing is a separate permission from reading and is billed separately. On a requester-pays bucket, RequestPayer must be passed to the list call as well as to the reads.

Example 3 — vector data over object storage 

import geopandas as gpd
import pyarrow.fs as fs


def read_parquet_s3(bucket, key, bbox=None, anonymous=False,
                    region="us-west-2"):
    """GeoParquet from S3, with a spatial filter where the file supports it."""
    filesystem = fs.S3FileSystem(region=region, anonymous=anonymous)
    path = f"{bucket}/{key}"

    try:
        gdf = gpd.read_parquet(path, filesystem=filesystem, bbox=bbox)
        print(f"  {len(gdf):,} rows with bbox pushdown")
    except ValueError as exc:
        if "bbox" not in str(exc):
            raise
        print(f"  no covering-bbox column ({exc}); reading all and clipping")
        gdf = gpd.read_parquet(path, filesystem=filesystem)
        if bbox:
            gdf = gdf.cx[bbox[0]:bbox[2], bbox[1]:bbox[3]]
        print(f"  {len(gdf):,} rows after clipping")
    return gdf

The bbox= argument only works on files written with write_covering_bbox=True. Without it, geopandas raises rather than silently reading everything — which is the right behaviour, and the fallback should be explicit.

For formats with no cloud story at all — Shapefile, GeoPackage — GDAL will read them over /vsis3, and it will read most of the file to do it. Convert to GeoParquet or FlatGeobuf first.

Explanation

Why requester-pays exists and what it means for you

Egress is the expensive part of object storage. A public dataset served for free costs its owner in proportion to how popular it is, which is an unbounded liability.

Requester-pays moves that cost to the reader. The data stays open — anyone may read it — and each reader pays for their own transfer.

The practical consequences are worth planning for. You need an account and credentials even for open data. Every read is billed, including reads of scenes you discard. And colocating compute in the same region matters: same-region transfer is typically free, cross-region is not.

That last point often reshapes an architecture: pre-filter aggressively on metadata, and run the processing in the bucket's region.

Why /vsis3 and /vsicurl are different

/vsicurl is a generic HTTP reader. It issues plain GET requests with range headers and no authentication. For a public object that is all you need.

/vsis3 understands the S3 API: it signs requests with your credentials, handles region redirects, and passes the requester-pays header. It is also the only one that can read a private object.

Using an https:// URL for a private bucket therefore fails with a 403 that looks like a permissions problem, when the real issue is that the request was never signed.

Why latency dominates small reads

A windowed read from a COG is a handful of range requests. Against local disk each costs microseconds; against an object store each costs a round trip.

At 100 ms latency, five requests is half a second, and the transfer time for 1.6 MB is negligible by comparison. This inverts the usual optimisation instinct: reducing the number of requests matters far more than reducing the bytes.

Hence the value of GDAL_DISABLE_READDIR_ON_OPEN (four fewer requests), of merging consecutive ranges, and of keeping a dataset handle open across many reads.

Why to check the format before blaming the network

Reading a Shapefile from S3 works and is slow, because Shapefile has no internal index or chunking — the reader fetches the .shp, the .dbf and the .shx, and the format offers no way to read part of them meaningfully.

The same applies to a striped GeoTIFF or an uncompressed GeoPackage. The network is not the problem; the format is. Convert once to a cloud-native format and every later read is cheap.

Five round trips costing 500 ms against 130 ms to transfer 1.6 MB, so latency dominates.
This inverts the usual instinct: cut requests, not bytes.

Edge cases or notes

  • AWS_NO_SIGN_REQUEST and AWS_REQUEST_PAYER are mutually exclusive.
  • Use s3:// not https:// for anything needing credentials.
  • Set the regional endpoint, or pay a redirect per request.
  • Listing is billed and permissioned separately from reading.
  • Set retries. Transient 500s are normal at scale.
  • Colocate compute with the bucket where egress is billed.
  • GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR removes probe requests that are also billed list calls.
  • Convert Shapefile and GeoPackage before serving them from object storage.

FAQ

How do I read a COG from S3 in Python?

rasterio.open("s3://bucket/key") inside a rasterio.Env with the appropriate access settings — AWS_NO_SIGN_REQUEST=YES for public buckets, credentials plus AWS_REQUEST_PAYER=requester for requester-pays.

Why does a public dataset need credentials?

Because it is on a requester-pays bucket, where the reader pays for transfer. The USGS Landsat Collection 2 archive refuses anonymous reads outright.

What is the difference between s3:// and https://?

s3:// uses GDAL's /vsis3, which signs requests and supports requester-pays. https:// uses /vsicurl, which does neither and works only for fully public objects.

Why are my reads slow even though the file is small?

Latency, not bandwidth. Each range request is a round trip, so reducing the number of requests matters more than reducing bytes.

Do I need to set the region?

Yes, or requests to the wrong endpoint incur a redirect — an extra round trip per request.

Can I read a Shapefile from S3?

GDAL will do it, and it will read most of the file. Convert to GeoParquet or FlatGeobuf first.

How do I avoid being billed for data I discard?

Filter on catalogue metadata before reading pixels, and colocate compute in the bucket's region where same-region transfer is free.