How to Read Spatial Data from S3 and Other Object Storage

On this page

Problem statement

Reading spatial data straight from object storage removes the download step, and adds a configuration surface that fails in ways local files never do: credentials, requester-pays, region endpoints, and per-request latency.

The failures are specific. A public dataset that needs credentials anyway:

RasterioIOError: AccessDenied: Anonymous users cannot invoke requests
against Requester Pays buckets. Please authenticate.

That is the USGS Landsat Collection 2 archive — open data on a requester-pays bucket, where anonymous reads are refused and the reader pays the transfer. Sentinel-2 on AWS, by contrast, is openly readable with no account at all.

Quick answer

import rasterio
from rasterio.windows import Window

window = Window(0, 0, 512, 512)

# open, no credentials
with rasterio.Env(AWS_NO_SIGN_REQUEST="YES", AWS_REGION="us-west-2",
                  GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR"):
    with rasterio.open(SENTINEL_S3) as ds:
        data = ds.read(1, window=window)
print("Sentinel-2:", data.shape, data.dtype)

# requester-pays: credentials AND the payer flag
try:
    with rasterio.Env(AWS_REQUEST_PAYER="requester", AWS_REGION="us-west-2",
                      GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR"):
        with rasterio.open(LANDSAT_S3) as ds:
            data = ds.read(1, window=window)
except rasterio.errors.RasterioIOError as exc:
    print("Landsat:", str(exc).split(". ")[0])

# and anonymously, as the Sentinel-2 read did
try:
    with rasterio.Env(AWS_NO_SIGN_REQUEST="YES", AWS_REGION="us-west-2"), rasterio.open(LANDSAT_S3):
        pass
except rasterio.errors.RasterioIOError as exc:
    print("Landsat, anonymous:", exc)

Sentinel-2: (512, 512) uint16
Landsat: InvalidCredentials: No valid AWS credentials found
Landsat, anonymous: AccessDenied: Anonymous users cannot invoke requests against Requester Pays buckets. Please authenticate.

The second read fails here because this machine has no AWS credentials, which is the point: with an account configured, the same code reads the Landsat band and the account is billed for the transfer.

AWS_NO_SIGN_REQUEST and AWS_REQUEST_PAYER are mutually exclusive: anonymous requests cannot be billed, so a requester-pays bucket rejects them regardless of the payer flag.

Open data is not the same as freely readable. Requester-pays needs an account before the first byte.

Access mode, path form, region, GDAL options, retries

1. Work out which access mode the bucket uses

mode	settings	example
public, anonymous	`AWS_NO_SIGN_REQUEST=YES`	Sentinel-2 COGs on AWS
public, requester-pays	credentials + `AWS_REQUEST_PAYER=requester`	USGS Landsat Collection 2
private	credentials	your own bucket

The middle row surprises people. The data is open, the licence is permissive, and you still need an AWS account because the storage owner does not pay for egress.

2. Use the right path form

"s3://bucket/key"                                   # GDAL /vsis3, needs config
"/vsis3/bucket/key"                                 # the same, explicit
"https://bucket.s3.amazonaws.com/key"               # /vsicurl, public only

The https:// form goes through /vsicurl, which does no signing — it works for public objects and fails on anything requiring credentials. The s3:// form uses /vsis3, which handles signing, regions and requester-pays.

Other providers have their own prefixes: /vsigs/ for Google Cloud Storage, /vsiaz/ for Azure Blob Storage, /vsioss/ for Alibaba.

3. Set the region, or pay for a redirect

"AWS_S3_ENDPOINT": "s3.us-west-2.amazonaws.com"

Requests to the wrong regional endpoint get a redirect. Counting the HTTP exchanges curl reports for one windowed read of the Sentinel-2 band, with and without the region set:

import subprocess
import sys

READ = f"""
import rasterio
from rasterio.windows import Window
with rasterio.open({SENTINEL_S3!r}) as ds:
    ds.read(1, window=Window(0, 0, 512, 512))
"""
for label, region in (("AWS_REGION=us-west-2", {"AWS_REGION": "us-west-2"}), ("no region", {})):
    env = {k: v for k, v in os.environ.items() if not k.startswith("AWS_REGION")}
    env |= {"AWS_NO_SIGN_REQUEST": "YES", "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
            "CPL_CURL_VERBOSE": "YES"} | region
    trace = subprocess.run([sys.executable, "-c", READ], env=env, capture_output=True, text=True).stderr
    statuses = [line.split()[2] for line in trace.splitlines() if line.startswith("< HTTP/")]
    print(f"{label:22} {len(statuses)} HTTP exchanges: {', '.join(statuses)}")

AWS_REGION=us-west-2   2 HTTP exchanges: 206, 206
no region              3 HTTP exchanges: 301, 206, 206

One extra round trip, not one per request: GDAL follows the 301 and remembers the right endpoint for the rest of the file. It is still a round trip on every file opened, which across a thousand scenes is a thousand of them.

4. Set the same GDAL options as for any remote read

"GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
"CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.TIF",
"VSI_CACHE": "TRUE",

Measured on a remote COG, GDAL_DISABLE_READDIR_ON_OPEN removed a probe for a sidecar file that did not exist — one request of seven with GDAL 3.12. On object storage GDAL's directory reading is a listing request, which is billed separately from GetObject.

5. Set retries

Object stores return transient 500s and 503s under load. Without retries, the first bad response a batch job meets fails it.

"GDAL_HTTP_MAX_RETRY": 3,
"GDAL_HTTP_RETRY_DELAY": 1,

The https form cannot sign a request. On a private or requester-pays bucket it can only fail.

Anonymous Sentinel-2 and requester-pays Landsat over Manchester

The data used on this page

Two real open archives on AWS: the Sentinel-2 L2A COGs in sentinel-cogs, readable by anyone, and the USGS Landsat Collection 2 archive in usgs-landsat, which is requester-pays. The setup finds one scene of each over Manchester through the public STAC catalogues — Earth Search for Sentinel-2, the USGS LandsatLook server for Landsat — and keeps the s3:// paths. Nothing is downloaded.

import os

import pystac_client

s2 = next(pystac_client.Client.open("https://earth-search.aws.element84.com/v1")
          .search(ids=["S2A_30UVD_20250922_0_L2A"], collections=["sentinel-2-l2a"]).items())
SENTINEL_HTTPS = s2.assets["red"].href
SENTINEL_S3 = "s3://" + SENTINEL_HTTPS.removeprefix("https://").replace(".s3.us-west-2.amazonaws.com", "")
landsat = next(pystac_client.Client.open("https://landsatlook.usgs.gov/stac-server")
               .search(collections=["landsat-c2l2-sr"], bbox=[-2.3, 53.4, -2.2, 53.5],
                       datetime="2025-06-01/2025-09-30", max_items=1).items())
LANDSAT_S3 = landsat.assets["red"].extra_fields["alternate"]["s3"]["href"]
print(SENTINEL_S3)
print(LANDSAT_S3)

s3://sentinel-cogs/sentinel-s2-l2a-cogs/30/U/VD/2025/9/S2A_30UVD_20250922_0_L2A/B04.tif
s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2025/203/023/LC09_L2SP_203023_20250926_20250927_02_T1/LC09_L2SP_203023_20250926_20250927_02_T1_SR_B4.TIF

Example 1 — configuration by bucket

import os
import rasterio

PROFILES = {
    "sentinel-cogs": {
        "AWS_NO_SIGN_REQUEST": "YES",
        "AWS_S3_ENDPOINT": "s3.us-west-2.amazonaws.com",
    },
    "usgs-landsat": {
        "AWS_REQUEST_PAYER": "requester",
        "AWS_S3_ENDPOINT": "s3.us-west-2.amazonaws.com",
    },
}

COMMON = {
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
    "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.TIF,.tiff,.jp2",
    "VSI_CACHE": "TRUE",
    "VSI_CACHE_SIZE": str(64 * 1024 * 1024),
    "GDAL_HTTP_MAX_RETRY": 3,
    "GDAL_HTTP_RETRY_DELAY": 1,
}


def env_for(url):
    """GDAL settings appropriate to the bucket in this URL."""
    bucket = url.split("/")[2] if url.startswith("s3://") else ""
    settings = dict(COMMON)
    settings.update(PROFILES.get(bucket, {}))

    if settings.get("AWS_REQUEST_PAYER") and not os.environ.get("AWS_ACCESS_KEY_ID"):
        raise RuntimeError(
            f"{bucket} is requester-pays and no AWS credentials are set. "
            "Anonymous reads are refused regardless of the payer flag.")
    return rasterio.Env(**settings)


def read_remote(url, **kwargs):
    with env_for(url):
        with rasterio.open(url) as ds:
            return ds.read(**kwargs)

print(read_remote(SENTINEL_S3, indexes=1, window=Window(0, 0, 256, 256)).shape)
try:
    read_remote(LANDSAT_S3, indexes=1, window=Window(0, 0, 256, 256))
except RuntimeError as exc:
    print(exc)

(256, 256)
usgs-landsat is requester-pays and no AWS credentials are set. Anonymous reads are refused regardless of the payer flag.

Raising before the read, with a message naming the bucket and the reason, turns a cryptic AccessDenied into an actionable one. On a requester-pays bucket the failure is otherwise indistinguishable from a permissions problem on your own data.

Example 2 — listing objects without downloading them

import boto3
from botocore import UNSIGNED
from botocore.config import Config


def list_prefix(bucket, prefix, requester_pays=False, anonymous=False, suffix=".tif", limit=50):
    """Enumerate objects and their sizes before deciding what to read."""
    client = boto3.client("s3", region_name="us-west-2",
                          config=Config(signature_version=UNSIGNED) if anonymous else None)
    kwargs = {"Bucket": bucket, "Prefix": prefix}
    if requester_pays:
        kwargs["RequestPayer"] = "requester"

    total_bytes, found = 0, []
    paginator = client.get_paginator("list_objects_v2")
    for page in paginator.paginate(**kwargs):
        for obj in page.get("Contents", []):
            if obj["Key"].endswith(suffix):
                found.append((obj["Key"], obj["Size"]))
                total_bytes += obj["Size"]
            if len(found) >= limit:
                break

    print(f"  {len(found)} objects, {total_bytes / 1e9:.2f} GB total")
    for key, size in found[:5]:
        print(f"    {size / 1e6:8.1f} MB  {key}")
    return found

scene_prefix = SENTINEL_S3.removeprefix("s3://sentinel-cogs/").rsplit("/", 1)[0] + "/"
found = list_prefix("sentinel-cogs", scene_prefix, anonymous=True)

  17 objects, 1.56 GB total
         0.4 MB  sentinel-s2-l2a-cogs/30/U/VD/2025/9/S2A_30UVD_20250922_0_L2A/AOT.tif
         6.0 MB  sentinel-s2-l2a-cogs/30/U/VD/2025/9/S2A_30UVD_20250922_0_L2A/B01.tif
       206.9 MB  sentinel-s2-l2a-cogs/30/U/VD/2025/9/S2A_30UVD_20250922_0_L2A/B02.tif
       212.9 MB  sentinel-s2-l2a-cogs/30/U/VD/2025/9/S2A_30UVD_20250922_0_L2A/B03.tif
       209.3 MB  sentinel-s2-l2a-cogs/30/U/VD/2025/9/S2A_30UVD_20250922_0_L2A/B04.tif

anonymous=True sends unsigned requests, which is what a public bucket needs from a machine with no credentials; the page's first version called boto3.client("s3"), which fails without them.

Listing is a separate permission from reading and is billed separately. On a requester-pays bucket, RequestPayer must be passed to the list call as well as to the reads.

Example 3 — vector data over object storage

import geopandas as gpd
import pyarrow.fs as fs


def read_parquet_s3(bucket, key, bbox=None, anonymous=False,
                    region="us-west-2"):
    """GeoParquet from S3, with a spatial filter where the file supports it."""
    filesystem = fs.S3FileSystem(region=region, anonymous=anonymous)
    path = f"{bucket}/{key}"

    try:
        gdf = gpd.read_parquet(path, filesystem=filesystem, bbox=bbox)
        print(f"  {len(gdf):,} rows with bbox pushdown")
    except ValueError as exc:
        if "bbox" not in str(exc):
            raise
        print(f"  no covering-bbox column ({exc}); reading all and clipping")
        gdf = gpd.read_parquet(path, filesystem=filesystem)
        if bbox:
            gdf = gdf.cx[bbox[0]:bbox[2], bbox[1]:bbox[3]]
        print(f"  {len(gdf):,} rows after clipping")
    return gdf

Overture's division areas for the latest release — several files of hundreds of megabytes each — filtered to the Rotterdam–Den Haag box:

import time

s3 = fs.S3FileSystem(region="us-west-2", anonymous=True)
releases = sorted(i.path for i in s3.get_file_info(fs.FileSelector("overturemaps-us-west-2/release/")))
prefix = f"{releases[-1]}/theme=divisions/type=division_area/"            # the newest release
files = sorted(i.path for i in s3.get_file_info(fs.FileSelector(prefix)) if i.path.endswith(".parquet"))
started, total = time.perf_counter(), 0
for path in files:
    bucket, key = path.split("/", 1)
    total += len(read_parquet_s3(bucket, key, bbox=(4.2, 51.85, 4.6, 52.15), anonymous=True))
print(f"{total:,} division areas from {len(files)} files in {time.perf_counter() - started:.0f} s")

  0 rows with bbox pushdown
  0 rows with bbox pushdown
  2 rows with bbox pushdown
  363 rows with bbox pushdown
  0 rows with bbox pushdown
  0 rows with bbox pushdown
  0 rows with bbox pushdown
  0 rows with bbox pushdown
365 division areas from 8 files in 31 s

The bbox= argument only works on files written with a covering bounding-box column — Overture's bbox struct is one. Without it, geopandas raises rather than silently reading everything — which is the right behaviour, and the fallback should be explicit.

For formats with no cloud story at all — Shapefile, GeoPackage — GDAL will read them over /vsis3, and it will read most of the file to do it. Convert to GeoParquet or FlatGeobuf first.

Measured in the quick answer and step 3.

Requester-pays, /vsis3 against /vsicurl, latency

Why requester-pays exists and what it means for you

Egress is the expensive part of object storage. A public dataset served for free costs its owner in proportion to how popular it is, which is an unbounded liability.

Requester-pays moves that cost to the reader. The data stays open — anyone may read it — and each reader pays for their own transfer.

The practical consequences are worth planning for. You need an account and credentials even for open data. Every read is billed, including reads of scenes you discard. And colocating compute in the same region matters: same-region transfer is typically free, cross-region is not.

That last point often reshapes an architecture: pre-filter aggressively on metadata, and run the processing in the bucket's region.

Why `/vsis3` and `/vsicurl` are different

/vsicurl is a generic HTTP reader. It issues plain GET requests with range headers and no authentication. For a public object that is all you need.

/vsis3 understands the S3 API: it signs requests with your credentials, handles region redirects, and passes the requester-pays header. It is also the only one that can read a private object.

Using an https:// URL for a private bucket therefore fails with a 403 that looks like a permissions problem, when the real issue is that the request was never signed.

Why latency dominates small reads

A windowed read from a COG is a handful of range requests. Against local disk each costs microseconds; against an object store each costs a round trip.

At 100 ms latency, five requests is half a second, while a 512 × 512 window of a COG is 0.6–2.5 MB — a fraction of a second on any reasonable connection. This inverts the usual optimisation instinct: reducing the number of requests matters far more than reducing the bytes.

Hence the value of GDAL_DISABLE_READDIR_ON_OPEN (four fewer requests), of merging consecutive ranges, and of keeping a dataset handle open across many reads.

Why to check the format before blaming the network

Reading a Shapefile from S3 works and is slow, because Shapefile has no internal index or chunking — the reader fetches the .shp, the .dbf and the .shx, and the format offers no way to read part of them meaningfully.

The same applies to a striped GeoTIFF or an uncompressed GeoPackage. The network is not the problem; the format is. Convert once to a cloud-native format and every later read is cheap.

This inverts the usual instinct: cut requests, not bytes.

Mutually exclusive flags, endpoints and listing

AWS_NO_SIGN_REQUEST and AWS_REQUEST_PAYER are mutually exclusive.
Use s3:// not https:// for anything needing credentials.
Set the regional endpoint, or pay a redirect per request.
Listing is billed and permissioned separately from reading.
Set retries. Transient 500s are normal at scale.
Colocate compute with the bucket where egress is billed.
GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR removes probe requests that are also billed list calls.
Convert Shapefile and GeoPackage before serving them from object storage.

Internal links

GDAL cannot open an S3 or HTTPS path — diagnosing the open failure
How to read a COG from a URL without downloading the whole file — the settings that matter for reads
Cloud-native geospatial explained — why the format decides the cost
Reading a COG from a URL is slow or downloads everything — when reads are slow rather than failing
Sentinel-2 or Landsat? Choosing a satellite imagery source — where the requester-pays example comes from
How to write partitioned GeoParquet and query it with filters — vector data on object storage
How to handle credentials and secrets in an automated GIS job — managing the keys
How to cache downloaded GIS data so you fetch it once — avoiding repeated billed reads

FAQ

How do I read a COG from S3 in Python?

rasterio.open("s3://bucket/key") inside a rasterio.Env with the appropriate access settings — AWS_NO_SIGN_REQUEST=YES for public buckets, credentials plus AWS_REQUEST_PAYER=requester for requester-pays.

Why does a public dataset need credentials?

Because it is on a requester-pays bucket, where the reader pays for transfer. The USGS Landsat Collection 2 archive refuses anonymous reads outright.

What is the difference between `s3://` and `https://`?

s3:// uses GDAL's /vsis3, which signs requests and supports requester-pays. https:// uses /vsicurl, which does neither and works only for fully public objects.

Why are my reads slow even though the file is small?

Latency, not bandwidth. Each range request is a round trip, so reducing the number of requests matters more than reducing bytes.

Do I need to set the region?

Yes, or requests to the wrong endpoint incur a redirect — an extra round trip per request.

Can I read a Shapefile from S3?

GDAL will do it, and it will read most of the file. Convert to GeoParquet or FlatGeobuf first.

How do I avoid being billed for data I discard?

Filter on catalogue metadata before reading pixels, and colocate compute in the bucket's region where same-region transfer is free.