🔧 Fix Cloud-native formats Files & formats Cloud-Native GIS · step 7 of 14 Updated

GDAL Cannot Open an S3 or HTTPS Path

Problem statement

rasterio.open("s3://...") raises, and the message rarely names the real cause. Five failures produce four similar-looking errors:

RasterioIOError: AccessDenied: Anonymous users cannot invoke requests
                 against Requester Pays buckets.
RasterioIOError: InvalidCredentials: No valid AWS credentials found.
RasterioIOError: HTTP response code: 404
RasterioIOError: '...' not recognized as a supported file format.

The first two are configuration. The third is usually a path. The fourth is often a permissions error returning an HTML error page that GDAL then tries to parse as a raster.

Quick answer

Work through the four checks in order:

import os
import rasterio


def check_access(url):
    print(f"  AWS_ACCESS_KEY_ID set: {bool(os.environ.get('AWS_ACCESS_KEY_ID'))}")
    print(f"  AWS_NO_SIGN_REQUEST:   {os.environ.get('AWS_NO_SIGN_REQUEST')}")
    print(f"  AWS_REQUEST_PAYER:     {os.environ.get('AWS_REQUEST_PAYER')}")
    print(f"  path scheme:           {url.split('://')[0]}")

    for label, env in (
        ("anonymous", {"AWS_NO_SIGN_REQUEST": "YES"}),
        ("credentialed", {}),
        ("requester-pays", {"AWS_REQUEST_PAYER": "requester"}),
    ):
        try:
            with rasterio.Env(**env, GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR"):
                with rasterio.open(url) as ds:
                    print(f"  {label:16} OK — {ds.width}x{ds.height}")
                    return label
        except Exception as exc:
            print(f"  {label:16} {type(exc).__name__}: {str(exc)[:70]}")
    return None

Trying all three access modes takes seconds and identifies which the bucket needs.

A decision tree from the error message to the cause: requester-pays, missing credentials, wrong path form, wrong region, or a non-raster response.
Four error messages, five causes. The tree is quicker than reading the message carefully.

Step-by-step solution

1. AccessDenied ... Requester Pays

The bucket bills the reader, and anonymous requests cannot be billed.

rasterio.Env(AWS_REQUEST_PAYER="requester")     # plus real credentials

Setting AWS_NO_SIGN_REQUEST=YES alongside it does not help — the two are mutually exclusive, and an unsigned request is refused before the payer flag is considered.

This is the state of the USGS Landsat Collection 2 archive: open data, requester-pays bucket, anonymous reads refused.

2. InvalidCredentials: No valid AWS credentials found

GDAL looked for credentials and found none. Either supply them, or declare that you do not need them:

rasterio.Env(AWS_NO_SIGN_REQUEST="YES")        # public bucket

The error text suggests setting credentials, which is misleading when the bucket is genuinely public. For a public object the fix is the opposite: tell GDAL not to sign.

3. HTTP response code: 404

Usually the path, not the permissions. Three common shapes:

"s3://bucket/key"              # correct
"s3://bucket//key"             # double slash: a different key
"https://bucket.s3.amazonaws.com/key"   # wrong region redirects or 404s

Verify the object exists with the provider's own tooling before debugging GDAL:

import boto3
boto3.client("s3").head_object(Bucket=bucket, Key=key)

A 403 from head_object on a key you can see in a listing means permissions; a 404 means the key is wrong.

4. not recognized as a supported file format

GDAL fetched something and could not parse it. Almost always the server returned an XML or HTML error document, and GDAL tried to open that.

import requests
r = requests.get(url.replace("s3://", "https://s3.amazonaws.com/"),
                 headers={"Range": "bytes=0-511"})
print(r.status_code, r.content[:200])

If the first bytes are <?xml or <html, you have an error page. Read it — it will name the real problem.

5. Wrong region 

rasterio.Env(AWS_S3_ENDPOINT="s3.us-west-2.amazonaws.com")

A request to the wrong regional endpoint either redirects, costing a round trip per request, or fails outright depending on the bucket's configuration.

An XML error document returned by the server and passed to GDAL, which reports an unsupported file format rather than the real error.
"Unsupported file format" usually means GDAL is trying to open an error message.

Code examples

Example 1 — a diagnostic that reads the actual response 

import os
import requests
import rasterio


def diagnose_open(url, region="us-east-1"):
    """Find out what the server is really returning."""
    if url.startswith("s3://"):
        _, _, bucket, *key_parts = url.split("/")
        key = "/".join(key_parts)
        http_url = f"https://{bucket}.s3.{region}.amazonaws.com/{key}"
    else:
        http_url = url

    response = requests.get(http_url, headers={"Range": "bytes=0-1023"},
                            timeout=30)
    print(f"  HTTP {response.status_code}")
    for header in ("content-type", "content-length", "content-range",
                   "x-amz-request-id"):
        if header in response.headers:
            print(f"    {header}: {response.headers[header]}")

    body = response.content[:400]
    if body.startswith(b"<?xml") or body.startswith(b"<html"):
        print("  ! the server returned a document, not raster data:")
        print("   ", body.decode("utf-8", "replace")[:300])
    elif body[:2] in (b"II", b"MM"):
        print("  looks like a TIFF (byte order "
              f"{'little' if body[:2] == b'II' else 'big'}-endian)")
    elif response.status_code == 206:
        print(f"  server supports range requests, {len(response.content)} bytes")
    else:
        print(f"  first bytes: {body[:16]!r}")
    return response

Fetching the first kilobyte with requests bypasses GDAL entirely and shows what is actually arriving. That single step resolves most of these failures, because the error document names the problem in plain English.

Example 2 — verifying credentials independently 

import boto3
from botocore.exceptions import ClientError, NoCredentialsError


def verify_s3(bucket, key, requester_pays=False):
    """Test access with boto3, whose errors are clearer than GDAL's."""
    client = boto3.client("s3")
    kwargs = {"Bucket": bucket, "Key": key}
    if requester_pays:
        kwargs["RequestPayer"] = "requester"

    try:
        head = client.head_object(**kwargs)
        print(f"  OK: {head['ContentLength'] / 1e6:.2f} MB, "
              f"{head.get('ContentType')}")
        return True
    except NoCredentialsError:
        print("  no credentials configured — set AWS_ACCESS_KEY_ID and "
              "AWS_SECRET_ACCESS_KEY, or use AWS_NO_SIGN_REQUEST for a "
              "public bucket")
    except ClientError as exc:
        code = exc.response["Error"]["Code"]
        messages = {
            "404": "the key does not exist — check the path",
            "NoSuchKey": "the key does not exist — check the path",
            "403": "access denied — check permissions or requester-pays",
            "AccessDenied": ("access denied — if the bucket is requester-pays, "
                             "pass RequestPayer='requester' and use real "
                             "credentials"),
            "PermanentRedirect": "wrong region — set AWS_S3_ENDPOINT",
        }
        print(f"  {code}: {messages.get(code, exc.response['Error']['Message'])}")
    return False

boto3 errors distinguish "the key does not exist" from "you may not read it", which GDAL collapses into similar messages.

Example 3 — configuration that fails early and clearly 

import os
import rasterio


def configured_env(url, mode="auto"):
    """Return a rasterio.Env for this URL, raising with a useful message."""
    settings = {
        "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
        "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.TIF,.tiff,.jp2",
        "GDAL_HTTP_MAX_RETRY": 3,
        "GDAL_HTTP_RETRY_DELAY": 1,
    }
    have_credentials = bool(os.environ.get("AWS_ACCESS_KEY_ID") or
                            os.path.exists(os.path.expanduser("~/.aws/credentials")))

    if mode == "anonymous":
        settings["AWS_NO_SIGN_REQUEST"] = "YES"
    elif mode == "requester-pays":
        if not have_credentials:
            raise RuntimeError(
                "requester-pays requires credentials; anonymous requests are "
                "refused before the payer flag is considered")
        settings["AWS_REQUEST_PAYER"] = "requester"
    elif mode == "auto":
        if not have_credentials:
            settings["AWS_NO_SIGN_REQUEST"] = "YES"
            print("  no credentials found — trying anonymous access")

    if url.startswith("https://") and mode != "anonymous":
        print("  ! an https:// path uses /vsicurl, which cannot sign requests. "
              "Use s3:// for private or requester-pays buckets.")
    return rasterio.Env(**settings)

The https:// warning is worth having. It is a silent downgrade: the path works, the request is unsigned, and the failure appears as a permissions error rather than as a configuration one.

Explanation

Why the error messages are so unhelpful

GDAL's /vsis3 layer reports what the HTTP layer told it, and S3 deliberately returns the same AccessDenied for "you may not read this" and "this does not exist" — revealing which would leak information about private buckets.

So one message covers several causes, and GDAL adds no context about which configuration it used. Testing the three access modes explicitly, as in the quick answer, resolves the ambiguity faster than reading the message.

Why an error page becomes "unsupported file format"

GDAL asks for the first bytes of the object. If the request fails, S3 returns an XML error document with an HTTP error status — but the bytes arrive.

GDAL then tries to identify a format from those bytes, fails, and reports that the file is not a supported format. The underlying problem was permissions or a missing key, and the message names neither.

Fetching the first kilobyte with an HTTP client and looking at it is the fastest way through this, because the error document says exactly what went wrong.

Why https:// silently downgrades

The https:// form routes through /vsicurl, a generic HTTP reader with no knowledge of S3. It cannot sign requests, cannot pass a requester-pays header, and cannot follow a region redirect intelligently.

For a fully public object it works. For anything else it fails with a 403, which looks like a permissions problem with your credentials — when in fact your credentials were never sent.

Use s3:// unless you specifically want unsigned HTTP access.

Why region matters

S3 buckets live in one region, and a request to a different regional endpoint gets a PermanentRedirect or a 400.

GDAL can follow redirects, at the cost of a round trip per request. On a windowed read of five requests that is five extra round trips, which on a 100 ms link is half a second added to every read.

Setting AWS_S3_ENDPOINT to the bucket's own region removes it.

GDAL's collapsed error messages against boto3's, which distinguish missing keys from permission failures.
Two lines of boto3 resolve most of these faster than reading the GDAL message carefully.

Edge cases or notes

  • AWS_NO_SIGN_REQUEST and AWS_REQUEST_PAYER cannot both apply.
  • "Unsupported file format" usually means an error document. Fetch the first bytes and read them.
  • https:// cannot sign requests. Use s3:// for private or requester-pays buckets.
  • S3 returns AccessDenied for missing keys too, deliberately.
  • Set AWS_S3_ENDPOINT to the bucket's region.
  • Test with boto3.head_object; its errors are more specific than GDAL's.
  • A double slash in a key is a different key.
  • Credentials can come from a file, environment or instance role — check all three before concluding they are absent.

FAQ

Why does GDAL say "no valid AWS credentials found" for a public bucket?

Because GDAL tries to sign by default. Set AWS_NO_SIGN_REQUEST=YES to tell it the object is public.

What does "Anonymous users cannot invoke requests against Requester Pays buckets" mean?

The bucket bills the reader, so requests must be signed with real credentials and carry AWS_REQUEST_PAYER=requester. Anonymous access is refused outright.

Why does GDAL say my COG is not a supported file format?

It is almost certainly trying to parse an XML error document returned by the server. Fetch the first kilobyte with an HTTP client and read it.

Should I use s3:// or https://?

s3:// for anything private or requester-pays — it signs requests. https:// works only for fully public objects.

Why do I get a 404 for an object I can see?

Check the exact key for double slashes and case, and check the region. S3 also returns access errors that look like 404s for private objects.

Do I need to set the region?

Yes. A request to the wrong regional endpoint redirects at best, costing a round trip per request, and fails at worst.

How do I test access outside GDAL?

boto3.client("s3").head_object(...). Its errors distinguish missing keys from permission failures, which GDAL's do not.