GRIB File Won't Open with cfgrib: Fixing the Common Errors

Problem statement

xr.open_dataset(path, engine="cfgrib") is the standard way to read GRIB weather data in Python, and with real files from NOAA it fails in four characteristic ways. A compressed file raises EOFError: No valid message found. A model output file with many levels raises DatasetBuildError: multiple values for unique key. An MRMS radar product opens but its only variable is called unknown, with no units. And the worst one raises nothing at all: a stale index file next to the GRIB makes cfgrib return one file's data with another file's timestamp.

This page reproduces each failure with real files from the morning Hurricane Beryl made landfall in Texas — MRMS rainfall accumulations and GFS model output — and fixes it.

Quick answer

Decompress first, never trust a sidecar index, split mixed files into hypercubes, and name MRMS fields from NSSL's table:

def open_grib(path, **filters):
    path = Path(path)
    if path.suffix == ".gz":                                      # EOFError otherwise
        out = path.with_suffix("")
        if not out.exists():
            with gzip.open(path) as src, open(out, "wb") as dst:
                shutil.copyfileobj(src, dst)
        path = out
    kw = {"indexpath": ""}                                        # no .idx file: no stale index
    if filters:
        kw["filter_by_keys"] = filters
    return xr.open_dataset(path, engine="cfgrib", decode_timedelta=True, backend_kwargs=kw)

ds = open_grib("data/" + MRMS_00Z + ".gz")
print(list(ds.data_vars), float(ds["unknown"].max()), str(ds.time.values)[:16])
print(open_grib("data/" + GFS, typeOfLevel="meanSea")["prmsl"].attrs["GRIB_name"])

['unknown'] 369.1000061035156 2024-07-09T00:00
Pressure reduced to MSL

Compressed files, multiple values and other cfgrib errors

1. EOFError: No valid message found — the file is compressed

MRMS products are published gzipped, and GRIB decoders do not look inside a gzip stream. They scan for the four bytes GRIB that start a message and find none:

for name in (MRMS_00Z + ".gz", MRMS_00Z):
    try:
        xr.open_dataset("data/" + name, engine="cfgrib", decode_timedelta=True, backend_kwargs={"indexpath": ""})
        print(f"{name[-12:]}: opened")
    except Exception as e:
        print(f"{name[-12:]}: {type(e).__name__}: {str(e)[:60]}")
    print("   first bytes:", Path("data", name).read_bytes()[:4])

000.grib2.gz: EOFError: No valid message found: 'data/MRMS_MultiSensor_QPE_24H_Pass2
   first bytes: b'\x1f\x8b\x08\x08'
000000.grib2: opened
   first bytes: b'GRIB'

\x1f\x8b is the gzip signature. Decompress — to a file or into memory — before opening.

2. DatasetBuildError: multiple values for unique key — the file holds several hypercubes

An xarray Dataset needs every variable on a shared set of coordinates. A GFS output file mixes mean-sea-level pressure, 2 m temperature, 40-odd pressure levels, soil layers, and both instantaneous and accumulated fields. cfgrib refuses to guess how to combine them and tells you which keys split them:

try:
    xr.open_dataset("data/" + GFS, engine="cfgrib", decode_timedelta=True, backend_kwargs={"indexpath": ""})
except Exception as e:
    print(type(e).__name__ + ":", str(e).splitlines()[0])
    print("\n".join(str(e).splitlines()[1:4]))
try:
    open_grib("data/" + GFS, typeOfLevel="surface")
except Exception as e:
    print("typeOfLevel=surface alone:", str(e).splitlines()[1:3])

DatasetBuildError: multiple values for unique key, try re-open the file with one of:
    filter_by_keys={'typeOfLevel': 'meanSea'}
    filter_by_keys={'typeOfLevel': 'hybrid'}
    filter_by_keys={'typeOfLevel': 'atmosphere'}
typeOfLevel=surface alone: ["    filter_by_keys={'stepType': 'instant', 'typeOfLevel': 'surface'}", "    filter_by_keys={'stepType': 'avg', 'typeOfLevel': 'surface'}"]

Filter on the keys it suggests, adding stepType where one level type mixes instantaneous, averaged and accumulated fields, or let cfgrib split the file for you:

parts = cfgrib.open_datasets("data/" + GFS, indexpath="", decode_timedelta=True)
print(f"{len(parts)} hypercubes; e.g.", [sorted(p.data_vars)[:4] for p in parts[:4]])
sfc = open_grib("data/" + GFS, typeOfLevel="surface", stepType="instant")
print("surface, instant:", sorted(sfc.data_vars)[:10])

48 hypercubes; e.g. [['refc', 'tcc'], ['tcc'], ['cwat', 'cwork', 'pwat', 'r'], ['tcc']]
surface, instant: ['SUNSD', 'cape', 'cfrzr', 'cicep', 'cin', 'cnwat', 'cpofp', 'cpr', 'crain', 'csnow']

3. The variable is called unknown — the parameter is in a local table

GRIB identifies a field by three numbers: discipline, category and parameter. ecCodes knows the WMO tables. MRMS uses discipline 209, which is reserved for local use and is not in ecCodes, so the name, units and missing values are lost. Read the three numbers and look them up in the table NSSL publishes:

with open("data/" + MRMS_00Z, "rb") as fh:
    h = eccodes.codes_grib_new_from_file(fh)
    ids = tuple(eccodes.codes_get(h, k) for k in ("discipline", "parameterCategory", "parameterNumber"))
    eccodes.codes_release(h)
table = pd.read_csv("data/UserTable_MRMS_v12.2.csv")
table["Discipline"] = pd.to_numeric(table.Discipline, errors="coerce")   # two "Not GRIB2" rows make it text
row = table[(table.Discipline == ids[0]) & (table.Category == ids[1]) & (table.Parameter == ids[2])].iloc[0]
print(ids, "->", row.Name, f"[{row.Unit}]; missing {row.Missing}, no coverage {row['No Coverage']}")

qpe = ds["unknown"].rename(row.Name)
qpe = qpe.where(qpe >= 0)                                          # -1 missing, -3 no radar coverage
print(f"{row.Name}: {int((ds['unknown'] < 0).sum()):,} cells flagged; max {float(qpe.max()):.1f} {row.Unit}")

(209, 6, 41) -> MultiSensor_QPE_24H_Pass2 [mm]; missing -1.0, no coverage -3.0
MultiSensor_QPE_24H_Pass2: 1,512,918 cells flagged; max 369.1 mm

Over 1.5 million cells hold a flag, not rain: the parts of the grid that no radar reaches. Without the table, a reader sees a field called unknown whose minimum is −3 and might average the flags into the rain.

4. No error, wrong answer — a stale index file

By default cfgrib writes an index next to the GRIB file (file.grib2.5b7b6.idx) and reuses it next time. It checks only that the index is newer than the file. Replace a file with a copy that keeps an older modification time — cp -p, rsync -a, unpacking an archive — and the old index is trusted:

work = Path("data/stale"); work.mkdir(exist_ok=True)
target = work / "qpe.grib2"
shutil.copy("data/" + MRMS_00Z, target)
first = xr.open_dataset(target, engine="cfgrib", decode_timedelta=True)            # writes the .idx
print("first open :", str(first.time.values)[:16], f"max {float(first['unknown'].max()):.1f} mm")
first.close()
shutil.copy2("data/" + MRMS_12Z, target)                            # replace, keeping an old mtime
os.utime(target, (1e9, 1e9))
second = xr.open_dataset(target, engine="cfgrib", decode_timedelta=True)
truth = open_grib("data/" + MRMS_12Z)
print("after swap :", str(second.time.values)[:16], f"max {float(second['unknown'].max()):.1f} mm")
print("really     :", str(truth.time.values)[:16], f"max {float(truth['unknown'].max()):.1f} mm")

first open : 2024-07-09T00:00 max 369.1 mm
after swap : 2024-07-09T00:00 max 232.2 mm
really     : 2024-07-09T12:00 max 232.2 mm

The values are the new file's; the time label is the old file's. Nothing warns. Pass indexpath="" — as open_grib does — or delete *.idx whenever files are replaced.

Reading only the messages you need

The data used on this page

MRMS MultiSensor QPE 24-hour Pass 2 ending 00 UTC and 12 UTC on 9 July 2024 from the NOAA MRMS archive on AWS; GFS 1.0° forecast from 00 UTC on 8 July 2024, 6 hours ahead, from the NOAA GFS archive on AWS; NSSL's MRMS GRIB2 table, version 12.2.

import gzip
import os
import shutil
from pathlib import Path

import cfgrib
import eccodes
import numpy as np
import pandas as pd
import requests
import xarray as xr

Path("data").mkdir(exist_ok=True)
MRMS = "https://noaa-mrms-pds.s3.amazonaws.com/CONUS/MultiSensor_QPE_24H_Pass2_00.00/"
MRMS_00Z = "MRMS_MultiSensor_QPE_24H_Pass2_00.00_20240709-000000.grib2"
MRMS_12Z = "MRMS_MultiSensor_QPE_24H_Pass2_00.00_20240709-120000.grib2"
GFS = "gfs.t00z.pgrb2.1p00.f006"
downloads = {
    MRMS_00Z + ".gz": MRMS + "20240709/" + MRMS_00Z + ".gz",
    MRMS_12Z + ".gz": MRMS + "20240709/" + MRMS_12Z + ".gz",
    GFS: "https://noaa-gfs-bdp-pds.s3.amazonaws.com/gfs.20240708/00/atmos/" + GFS,
    "UserTable_MRMS_v12.2.csv": "https://raw.githubusercontent.com/NOAA-National-Severe-Storms-Laboratory/"
                                "mrms-support/main/GRIB2_TABLES/UserTable_MRMS_v12.2.csv",
}
for name, url in downloads.items():
    if not Path("data", name).exists():
        Path("data", name).write_bytes(requests.get(url, timeout=600).content)
with gzip.open("data/" + MRMS_12Z + ".gz") as src, open("data/" + MRMS_12Z, "wb") as dst:
    shutil.copyfileobj(src, dst)
for p in Path("data").glob("*.idx"):
    p.unlink()
print({n: f"{Path('data', n).stat().st_size / 1e6:.1f} MB" for n in downloads})

{'MRMS_MultiSensor_QPE_24H_Pass2_00.00_20240709-000000.grib2.gz': '4.4 MB', 'MRMS_MultiSensor_QPE_24H_Pass2_00.00_20240709-120000.grib2.gz': '4.4 MB', 'gfs.t00z.pgrb2.1p00.f006': '45.1 MB', 'UserTable_MRMS_v12.2.csv': '0.0 MB'}

Example 1: read only the messages you need

A GFS 0.25° file is over half a gigabyte, but every file on the archive has a text inventory, also called .idx (a different thing from cfgrib's index), listing where each message starts. Download just the byte ranges you want:

URL = "https://noaa-gfs-bdp-pds.s3.amazonaws.com/gfs.20240708/00/atmos/gfs.t00z.pgrb2.0p25.f009"
inventory = [line.split(":") for line in requests.get(URL + ".idx", timeout=60).text.splitlines()]
total = int(requests.head(URL, timeout=60).headers["Content-Length"])
wanted = {("PRMSL", "mean sea level"), ("PWAT", "entire atmosphere (considered as a single layer)")}
chunks = []
for i, rec in enumerate(inventory):
    if (rec[3], rec[4]) in wanted:
        end = int(inventory[i + 1][1]) - 1 if i + 1 < len(inventory) else total - 1
        chunks.append(requests.get(URL, headers={"Range": f"bytes={rec[1]}-{end}"}, timeout=120).content)
Path("data/gfs_subset.grib2").write_bytes(b"".join(chunks))
size = Path("data/gfs_subset.grib2").stat().st_size
print(f"{len(inventory)} messages, {total / 1e6:.0f} MB; downloaded {len(chunks)} = {size / 1e6:.2f} MB ({size / total:.1%})")

mslp = open_grib("data/gfs_subset.grib2", shortName="prmsl")["prmsl"].sel(latitude=slice(35, 20), longitude=slice(260, 275)) / 100
low = mslp.where(mslp == mslp.min(), drop=True)
print(f"GFS +9 h, valid {str(mslp.valid_time.values)[:16]}: minimum {float(mslp.min()):.1f} hPa at "
      f"{float(low.latitude[0]):.2f}N {360 - float(low.longitude[0]):.2f}W")

743 messages, 537 MB; downloaded 2 = 2.27 MB (0.4%)
GFS +9 h, valid 2024-07-08T09:00: minimum 977.9 hPa at 28.75N 96.00W

IBTrACS, the best-track archive, puts Beryl at 28.7°N 96.0°W at 09 UTC, with 978 hPa at its 08:40 UTC landfall. The model's nine-hour forecast has the centre within one grid cell.

Example 2: list what is in a file before opening it

with open("data/" + GFS, "rb") as fh:
    rows = []
    while (h := eccodes.codes_grib_new_from_file(fh)) is not None:
        rows.append({k: eccodes.codes_get(h, k) for k in ("shortName", "typeOfLevel", "stepType")})
        eccodes.codes_release(h)
inv = pd.DataFrame(rows)
print(f"{len(inv)} messages, {inv.typeOfLevel.nunique()} level types, {inv.stepType.nunique()} step types")
print(inv.groupby("typeOfLevel").size().sort_values(ascending=False).head(6).to_string())

743 messages, 35 level types, 5 step types
typeOfLevel
isobaricInhPa              462
isobaricInPa                80
surface                     56
heightAboveGround           27
pressureFromGroundLayer     12
depthBelowLandLayer         12

A GRIB file is a list of independent messages

A GRIB file is a sequence of independent messages, each one 2D field with its own metadata. It has no file header, no shared coordinates and no promise that the messages fit together. cfgrib's job is to group messages into an xarray hypercube, and it fails loudly when they do not fit, which is the DatasetBuildError. It also builds an index, because scanning a big file is slow. The index is the one part that can fail silently.

The name of a field is not stored in the file. Only its code numbers are, and the decoder looks them up in tables. That works for WMO and ECMWF parameters and fails for any centre's local ones. MRMS, the NCEP local tables and some ensemble products all need the producer's table. Compression and inventories are conventions of the archive, not of GRIB. The .gz must come off before decoding, and the .idx inventory lets you skip the decoder's scan altogether.

0-360 longitudes, index files and pygrib

• Longitudes run 0–360 in both MRMS and GFS; subtract 360 for the western hemisphere or sel will find nothing.
• Latitude is descending in both; slice from north to south.
• Read-only directories: cfgrib cannot write its index and warns; indexpath="" avoids the attempt.
• Two meanings of .idx: NOAA's text inventory and cfgrib's binary index share an extension; never mix them up in the same folder.
• pygrib and wgrib2 use the same tables and hit the same unknown problem with local parameters.
• GRIB1 files use different keys (indicatorOfParameter, table2Version).

Internal links

• How to read MRMS rainfall in Python — the product used here
• How to estimate rainfall from radar — comparing radar with MRMS
• How to map storm-total rainfall — MRMS totals by county
• How to plot a hurricane track from IBTrACS — the best-track check
• GRIB, NetCDF and Zarr compared — when GRIB is the right format

FAQ

Why does cfgrib say "No valid message found"?

The file is not raw GRIB, usually because it is gzipped (it starts with the bytes \x1f\x8b). Decompress it first.

How do I fix "multiple values for unique key"?

Pass filter_by_keys with the keys cfgrib suggests, such as typeOfLevel and stepType, or use cfgrib.open_datasets to get one Dataset per hypercube.

Why is my GRIB variable called "unknown"?

The parameter is in a local table that ecCodes does not have. For MRMS, look up discipline 209, the category and the parameter number in NSSL's GRIB2 table.

Can cfgrib return the wrong data without an error?

Yes. If a GRIB file is replaced by one with an older modification time, cfgrib trusts the old index file and labels the new data with the old time. Use indexpath="".