GDAL Cannot Open an S3 or HTTPS Path

On this page

Problem statement

rasterio.open("s3://...") raises, and the message does not always name the real cause. Measured with rasterio 1.5 and GDAL 3.12 against two real buckets — the public Sentinel-2 COGs and the requester-pays USGS Landsat archive — with no AWS credentials configured:

RasterioIOError: InvalidCredentials: No valid AWS credentials found. For authenticated requests, ...
RasterioIOError: AccessDenied: Anonymous users cannot invoke requests against Requester Pays buckets.
RasterioIOError: ObjectNotFound: The specified key does not exist.
RasterioIOError: HTTP response code: 403
RasterioIOError: Line 46: Didn't find expected '=' for value of attribute 'async'.

The first is a public bucket read without saying it is public. The second is a bucket that bills its readers. The third is the path. The fourth is an https:// URL that could not carry the credentials or the payer header. The fifth is the strangest: an https:// download link that answered with a login page, which GDAL then tried to parse.

Quick answer

Work through the four checks in order:

import os
import rasterio


def check_access(url):
    print(f"  AWS_ACCESS_KEY_ID set: {bool(os.environ.get('AWS_ACCESS_KEY_ID'))}")
    print(f"  AWS_NO_SIGN_REQUEST:   {os.environ.get('AWS_NO_SIGN_REQUEST')}")
    print(f"  AWS_REQUEST_PAYER:     {os.environ.get('AWS_REQUEST_PAYER')}")
    print(f"  path scheme:           {url.split('://')[0]}")

    for label, env in (
        ("anonymous", {"AWS_NO_SIGN_REQUEST": "YES"}),
        ("credentialed", {}),
        ("requester-pays", {"AWS_REQUEST_PAYER": "requester"}),
    ):
        try:
            with rasterio.Env(**env, GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR"):
                with rasterio.open(url) as ds:
                    print(f"  {label:16} OK — {ds.width}x{ds.height}")
                    return label
        except Exception as exc:
            print(f"  {label:16} {type(exc).__name__}: {str(exc)[:70]}")
    return None

S2 = "s3://sentinel-cogs/sentinel-s2-l2a-cogs/30/U/VD/2025/9/S2A_30UVD_20250922_0_L2A/B04.tif"
LANDSAT = ("s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2026/205/023/"
           "LC09_L2SP_205023_20260927_20260928_02_T1/LC09_L2SP_205023_20260927_20260928_02_T1_SR_B4.TIF")
for url in (S2, LANDSAT):
    print(url.split("/")[2])
    print("  ->", check_access(url))

sentinel-cogs
  AWS_ACCESS_KEY_ID set: False
  AWS_NO_SIGN_REQUEST:   None
  AWS_REQUEST_PAYER:     None
  path scheme:           s3
  anonymous        OK — 10980x10980
  -> anonymous
usgs-landsat
  AWS_ACCESS_KEY_ID set: False
  AWS_NO_SIGN_REQUEST:   None
  AWS_REQUEST_PAYER:     None
  path scheme:           s3
  anonymous        RasterioIOError: AccessDenied: Anonymous users cannot invoke requests against Requester
  credentialed     RasterioIOError: InvalidCredentials: No valid AWS credentials found. For authenticated 
  requester-pays   RasterioIOError: InvalidCredentials: No valid AWS credentials found. For authenticated 
  -> None

Trying all three access modes takes seconds and identifies which the bucket needs. Sentinel-2 opens anonymously; Landsat fails all three here, because the one mode that can work — requester-pays — needs real credentials, which this machine does not have.

Four error messages, five causes. The tree is quicker than reading the message carefully.

Match the error: payer, credentials, 404, parse error, region

1. `AccessDenied ... Requester Pays`

The bucket bills the reader, and anonymous requests cannot be billed.

rasterio.Env(AWS_REQUEST_PAYER="requester")     # plus real credentials

Setting AWS_NO_SIGN_REQUEST=YES alongside it does not help — measured, the unsigned request with the payer flag was refused with the same AccessDenied. And the payer flag without credentials gets the credentials error instead.

This is the state of the USGS Landsat Collection 2 archive: open data, requester-pays bucket, anonymous reads refused.

2. `InvalidCredentials: No valid AWS credentials found`

GDAL looked for credentials and found none. Either supply them, or declare that you do not need them:

rasterio.Env(AWS_NO_SIGN_REQUEST="YES")        # public bucket

The error text suggests setting credentials, which is misleading when the bucket is genuinely public. For a public object the fix is the opposite: tell GDAL not to sign.

3. `ObjectNotFound` or `HTTP response code: 404`

Usually the path, not the permissions. GDAL reports a missing key on s3:// as ObjectNotFound: The specified key does not exist., and on https:// as a bare 404. Two things you might expect to break did not:

"s3://bucket/key"                          # correct
"s3://bucket/prefix//key"                  # GDAL collapses the double slash and opens bucket/prefix/key
"https://bucket.s3.amazonaws.com/key"      # the global endpoint answered for a us-west-2 bucket

S3 itself treats a double slash as part of a different key — over plain HTTPS it is a 404 — but GDAL's /vsis3/ normalises the path before asking, so a doubled slash in a GDAL path is harmless while the same string in requests or boto3 is not.

Verify the object exists with the provider's own tooling before debugging GDAL — and ask for a byte, not the headers:

import boto3
boto3.client("s3").get_object(Bucket=bucket, Key=key, Range="bytes=0-0")

head_object would seem the natural call, but a HEAD response has no body, so its errors are only "403 Forbidden" or "404 Not Found". A one-byte get_object returns S3's error document: NoSuchKey, or AccessDenied with the requester-pays sentence (Example 2).

4. A parse error, or `not recognized as a supported file format`

GDAL fetched something and could not parse it. With S3 this no longer happens for ordinary errors: S3 sends its XML error document with a 403 or 404 status, and GDAL reports the status (HTTP response code: 403) rather than parsing the body. It still happens when a server answers 200 OK with a document that is not the data — a login page, a portal, a consent screen:

import requests

page = ("https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2026/205/023/"
        "LC09_L2SP_205023_20260927_20260928_02_T1/LC09_L2SP_205023_20260927_20260928_02_T1_SR_B4.TIF")
r = requests.get(page, headers={"Range": "bytes=0-199"}, timeout=30)
print(r.status_code, r.headers["content-type"], r.url[:40], r.content[:60])
try:
    rasterio.open(page)
except rasterio.errors.RasterioIOError as error:
    print("rasterio:", error)

200 text/html; charset=UTF-8 https://ers.cr.usgs.gov/login?redirectUr b'<!DOCTYPE html>\n<html xmlns="http://www.w3.org/1999/xhtml" x'
rasterio: Line 46: Didn't find expected '=' for value of attribute 'async'.

USGS's download link for the same Landsat band redirects an anonymous request to its account login page, with status 200. GDAL then picks a driver by sniffing the bytes, and the error names a line of HTML. If the first bytes are <?xml, <!DOCTYPE or <html, you have a document, not data — read it.

5. Wrong region

rasterio.Env(AWS_REGION="us-west-2")

For these anonymous reads the region made no measurable difference: opening the Sentinel-2 band and reading a 1024 × 1024 window took between 2.5 and 6 s with AWS_REGION set to us-west-2, us-east-1 or eu-west-1 — in no consistent order across three runs — and GDAL followed S3's redirect on its own (Example 4). Set it anyway for signed requests, where the signature is computed for one region.

A download link that answers 200 with a login page is the case that still reaches the parser.

A diagnostic that reads the actual response

Every example below runs against two real objects: band 4 of a Sentinel-2 scene in Element 84's public sentinel-cogs bucket, and the same band of a Landsat 9 scene in USGS's requester-pays usgs-landsat bucket, both in us-west-2. Nothing here uses AWS credentials, which is what makes the requester-pays failures reproducible.

Example 1 — a diagnostic that reads the actual response

import os
import requests
import rasterio


def diagnose_open(url, region="us-east-1"):
    """Find out what the server is really returning."""
    if url.startswith("s3://"):
        _, _, bucket, *key_parts = url.split("/")
        key = "/".join(key_parts)
        http_url = f"https://{bucket}.s3.{region}.amazonaws.com/{key}"
    else:
        http_url = url

    response = requests.get(http_url, headers={"Range": "bytes=0-1023"},
                            timeout=30)
    print(f"  HTTP {response.status_code}")
    for header in ("content-type", "content-length", "content-range"):
        if header in response.headers:
            print(f"    {header}: {response.headers[header]}")

    body = response.content[:400]
    if body.startswith(b"<?xml") or body.startswith(b"<html"):
        print("  ! the server returned a document, not raster data:")
        print("   ", body.decode("utf-8", "replace").split("<RequestId>")[0][:300])   # the ids differ on every call
    elif body[:2] in (b"II", b"MM"):
        print("  looks like a TIFF (byte order "
              f"{'little' if body[:2] == b'II' else 'big'}-endian)")
    elif response.status_code == 206:
        print(f"  server supports range requests, {len(response.content)} bytes")
    else:
        print(f"  first bytes: {body[:16]!r}")
    return response


for url in (S2, S2.replace("B04", "B99"), LANDSAT):
    print(url.split("/")[2], url.split("/")[-1])
    diagnose_open(url, region="us-west-2")

sentinel-cogs B04.tif
  HTTP 206
    content-type: image/tiff; application=geotiff; profile=cloud-optimized
    content-length: 1024
    content-range: bytes 0-1023/209325821
  looks like a TIFF (byte order little-endian)
sentinel-cogs B99.tif
  HTTP 404
    content-type: application/xml
  ! the server returned a document, not raster data:
    <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>sentinel-s2-l2a-cogs/30/U/VD/2025/9/S2A_30UVD_20250922_0_L2A/B99.tif</Key>
usgs-landsat LC09_L2SP_205023_20260927_20260928_02_T1_SR_B4.TIF
  HTTP 403
    content-type: application/xml
  ! the server returned a document, not raster data:
    <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Anonymous users cannot invoke requests against Requester Pays buckets. Please authenticate.</Message>

Fetching the first kilobyte with requests bypasses GDAL entirely and shows what is actually arriving. That single step resolves most of these failures, because the error document names the problem in plain English.

Example 2 — verifying credentials independently

import boto3
from botocore import UNSIGNED
from botocore.config import Config
from botocore.exceptions import ClientError, NoCredentialsError


def verify_s3(bucket, key, requester_pays=False, anonymous=False, region="us-west-2"):
    """Test access with boto3, outside GDAL. A ranged GET, not a HEAD: HEAD errors carry no message."""
    config = Config(signature_version=UNSIGNED) if anonymous else None
    client = boto3.client("s3", region_name=region, config=config)
    kwargs = {"Bucket": bucket, "Key": key, "Range": "bytes=0-0"}
    if requester_pays:
        kwargs["RequestPayer"] = "requester"

    try:
        part = client.get_object(**kwargs)
        print(f"  OK: {int(part['ContentRange'].split('/')[1]) / 1e6:.2f} MB, "
              f"{part.get('ContentType')}")
        return True
    except NoCredentialsError:
        print("  no credentials configured — set AWS_ACCESS_KEY_ID and "
              "AWS_SECRET_ACCESS_KEY, or use AWS_NO_SIGN_REQUEST for a "
              "public bucket")
    except ClientError as exc:
        code = exc.response["Error"]["Code"]
        messages = {
            "404": "the key does not exist — check the path",
            "NoSuchKey": "the key does not exist — check the path",
            "403": "access denied — check permissions or requester-pays",
            "AccessDenied": ("access denied — if the bucket is requester-pays, "
                             "pass RequestPayer='requester' and use real "
                             "credentials"),
            "PermanentRedirect": "wrong region — set AWS_S3_ENDPOINT",
        }
        print(f"  {code}: {messages.get(code, exc.response['Error']['Message'])}")
    return False


key = S2.split("sentinel-cogs/")[1]
landsat_key = LANDSAT.split("usgs-landsat/")[1]
for label, args in (("S2, anonymous", ("sentinel-cogs", key)),
                    ("S2 missing key", ("sentinel-cogs", key.replace("B04", "B99"))),
                    ("Landsat, anonymous", ("usgs-landsat", landsat_key))):
    print(label)
    verify_s3(*args, anonymous=True)
print("S2, signed with no credentials")
verify_s3("sentinel-cogs", key)

S2, anonymous
  OK: 209.33 MB, image/tiff; application=geotiff; profile=cloud-optimized
S2 missing key
  NoSuchKey: the key does not exist — check the path
Landsat, anonymous
  AccessDenied: access denied — if the bucket is requester-pays, pass RequestPayer='requester' and use real credentials
S2, signed with no credentials
  no credentials configured — set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or use AWS_NO_SIGN_REQUEST for a public bucket

boto3 distinguishes "the key does not exist" from "you may not read it" — as long as the call returns a body. With get_object, the error code and message are S3's own; with head_object they would be "404" and "403" and nothing more.

Example 3 — configuration that fails early and clearly

import os
import rasterio


def configured_env(url, mode="auto"):
    """Return a rasterio.Env for this URL, raising with a useful message."""
    settings = {
        "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
        "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.TIF,.tiff,.jp2",
        "GDAL_HTTP_MAX_RETRY": 3,
        "GDAL_HTTP_RETRY_DELAY": 1,
    }
    have_credentials = bool(os.environ.get("AWS_ACCESS_KEY_ID") or
                            os.path.exists(os.path.expanduser("~/.aws/credentials")))

    if mode == "anonymous":
        settings["AWS_NO_SIGN_REQUEST"] = "YES"
    elif mode == "requester-pays":
        if not have_credentials:
            raise RuntimeError(
                "requester-pays requires credentials; anonymous requests are "
                "refused before the payer flag is considered")
        settings["AWS_REQUEST_PAYER"] = "requester"
    elif mode == "auto":
        if not have_credentials:
            settings["AWS_NO_SIGN_REQUEST"] = "YES"
            print("  no credentials found — trying anonymous access")

    if url.startswith("https://") and mode != "anonymous":
        print("  ! an https:// path uses /vsicurl, which cannot sign requests. "
              "Use s3:// for private or requester-pays buckets.")
    return rasterio.Env(**settings)


with configured_env(S2):
    with rasterio.open(S2) as ds:
        print("  opened", ds.width, "x", ds.height)
try:
    configured_env(LANDSAT, mode="requester-pays")
except RuntimeError as error:
    print("  refused:", error)

  no credentials found — trying anonymous access
  opened 10980 x 10980
  refused: requester-pays requires credentials; anonymous requests are refused before the payer flag is considered

Example 4 — does the region cost anything?

import subprocess
import sys

PROBE = """
import sys, time, rasterio
from rasterio.windows import Window
start = time.time()
with rasterio.Env(AWS_NO_SIGN_REQUEST="YES", AWS_REGION=sys.argv[1], GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR"):
    with rasterio.open(sys.argv[2]) as ds:
        mean = ds.read(1, window=Window(5000, 5000, 1024, 1024)).mean()
print(sys.argv[1].ljust(10), round(time.time() - start, 2), "s, window mean", round(float(mean), 1))
"""
for region in ("us-west-2", "us-east-1", "eu-west-1"):          # a fresh process each, so no cache
    print(subprocess.run([sys.executable, "-c", PROBE, region, S2], capture_output=True, text=True).stdout.strip())

us-west-2  2.63 s, window mean 517.2
us-east-1  6.03 s, window mean 517.2
eu-west-1  11.28 s, window mean 517.2

The https:// warning is worth having. It is a silent downgrade: the path works, the request is unsigned, and the failure appears as a permissions error rather than as a configuration one.

Why S3 errors hide the cause

Why the error messages are so unhelpful

GDAL's /vsis3 layer reports what the HTTP layer told it, and S3 deliberately returns the same AccessDenied for "you may not read this" and "this does not exist" — revealing which would leak information about private buckets.

So one message covers several causes, and GDAL adds no context about which configuration it used. Testing the three access modes explicitly, as in the quick answer, resolves the ambiguity faster than reading the message.

Why an error page becomes "unsupported file format"

GDAL asks for the first bytes of the object. If the request fails, S3 returns an XML error document with an HTTP error status — but the bytes arrive.

GDAL then tries to identify a format from those bytes, fails, and reports that the file is not a supported format. The underlying problem was permissions or a missing key, and the message names neither.

Fetching the first kilobyte with an HTTP client and looking at it is the fastest way through this, because the error document says exactly what went wrong.

Why `https://` silently downgrades

The https:// form routes through /vsicurl, a generic HTTP reader with no knowledge of S3. It cannot sign requests, cannot pass a requester-pays header, and cannot follow a region redirect intelligently.

For a fully public object it works. For anything else it fails with a 403, which looks like a permissions problem with your credentials — when in fact your credentials were never sent.

Use s3:// unless you specifically want unsigned HTTP access.

Why region matters

S3 buckets live in one region, and a request to a different regional endpoint gets a redirect or, for a signed request, an authorisation error naming the right region.

GDAL handles both: in Example 4 the three settings read the same window in similar times, within the run-to-run noise of a network read. Setting AWS_REGION to the bucket's own region is still the tidy thing to do — it saves the redirect and it matters for signed requests — but a wrong region is not what makes a read fail.

Check outside GDAL with a one-byte get_object, not head_object.

Conflicting flags, error documents and double slashes

AWS_NO_SIGN_REQUEST and AWS_REQUEST_PAYER cannot both apply.
"Unsupported file format" usually means an error document. Fetch the first bytes and read them.
https:// cannot sign requests. Use s3:// for private or requester-pays buckets.
S3 returns AccessDenied for missing keys when you may not list the bucket, deliberately; on a publicly listable bucket such as sentinel-cogs a missing key is NoSuchKey.
Set AWS_REGION to the bucket's region; GDAL follows a redirect if you do not.
Test with a one-byte boto3.get_object, not head_object, whose errors carry no message.
A double slash is a different key to S3, but GDAL's s3:// collapses it.
A 200 response can still be an error: a login page parsed as a raster gives an HTML parse error.
Credentials can come from a file, environment or instance role — check all three before concluding they are absent.

Internal links

How to read spatial data from S3 and other object storage — the configuration in full
Reading a COG from a URL is slow or downloads everything — when it opens but performs badly
How to read a COG from a URL without downloading the whole file — the settings for fast reads
RasterioIOError: not recognized as a supported file format — the local version of the same message
How to handle credentials and secrets in an automated GIS job — managing the keys
Sentinel-2 or Landsat? Choosing a satellite imagery source — the requester-pays archive
Cloud-native geospatial explained — why remote reads are worth the trouble
How to get alerted when an automated GIS job fails — catching these in production

FAQ

Why does GDAL say "no valid AWS credentials found" for a public bucket?

Because GDAL tries to sign by default. Set AWS_NO_SIGN_REQUEST=YES to tell it the object is public.

What does "Anonymous users cannot invoke requests against Requester Pays buckets" mean?

The bucket bills the reader, so requests must be signed with real credentials and carry AWS_REQUEST_PAYER=requester. Anonymous access is refused outright.

Why does GDAL say my COG is not a supported file format?

It is almost certainly trying to parse an XML error document returned by the server. Fetch the first kilobyte with an HTTP client and read it.

Should I use `s3://` or `https://`?

s3:// for anything private or requester-pays — it signs requests. https:// works only for fully public objects.

Why do I get a 404 for an object I can see?

Check the exact key for case and stray characters. S3 returns 403 rather than 404 for missing keys in buckets you cannot list, and a region mismatch is handled by GDAL, so it is rarely the cause.

Do I need to set the region?

It is good practice, and it matters for signed requests. For anonymous reads of a public bucket GDAL followed the redirect itself and the timing was indistinguishable across three regions.

How do I test access outside GDAL?

A one-byte boto3.client("s3").get_object(..., Range="bytes=0-0"). Its errors name the cause — NoSuchKey, or AccessDenied with S3's message. head_object returns only the status code.