⚙️ How-To Gridded & climate data Files & formats Data sources & downloads Gridded & Climate Data · step 14 of 20 Updated

How to Read GRIB Weather Forecast Data in Python

Problem statement

Weather forecasts from NOAA's GFS, ECMWF and most national services arrive as GRIB2: a file of hundreds of independent messages, each one field at one level and one forecast step. xarray reads GRIB through the cfgrib engine, which has to assemble those messages into a rectangular dataset — and a single GFS file does not fit into one. The result is a DatasetBuildError, a dataset that silently leaves out most of the variables you asked for, or precipitation whose meaning depends on a key you never looked at.

Measured on the GFS forecast of 10 September 2026, 00 UTC, with cfgrib 0.9.15 and ecCodes 2.48:

  • The 1° analysis file held 696 messages. Opened without a filter it raised DatasetBuildError: multiple values for unique key; cfgrib.open_datasets split it into 35 datasets with 139 variables.
  • Filtering on typeOfLevel="heightAboveGround" alone returned one variable — radar reflectivity — and logged that it had skipped 12 others, including 2 m temperature and 10 m wind.
  • Four fields from the 508 MB 0.25° file — 2 m temperature, 10 m wind and sea-level pressure — downloaded by byte range were 3.43 MB, 0.68% of the file, in 3.98 seconds.
  • The 6-hour surface file mixed instantaneous, averaged and accumulated fields: precipitation rate was an average over the step, total precipitation an accumulation, and neither opened until stepType was in the filter.

Quick answer

Open one kind of level at a time with filter_by_keys, and add stepType for forecast steps:

import xarray as xr

t2m = xr.open_dataset(
    "gfs.t00z.pgrb2.1p00.f000", engine="cfgrib",
    backend_kwargs={"filter_by_keys": {"typeOfLevel": "heightAboveGround", "level": 2}, "indexpath": ""},
)["t2m"]
print(dict(t2m.sizes), t2m.attrs["GRIB_units"], float(t2m.sel(latitude=51.0, longitude=0.0)) - 273.15)
{'latitude': 181, 'longitude': 360} K 13.772546386718773

typeOfLevel separates pressure levels, heights, the surface and so on; level or shortName narrows further; stepType separates instantaneous, averaged and accumulated fields. indexpath="" stops cfgrib writing an index file beside the data.

Flow from a GRIB file of independent messages through filter_by_keys on typeOfLevel, level and stepType to a rectangular xarray Dataset.
A GRIB file is a list of messages; the filter decides which of them can share one set of coordinates.

Step-by-step solution

1. Install cfgrib and ecCodes

cfgrib is the xarray engine; ecCodes is the ECMWF library that decodes GRIB. pip install cfgrib eccodes installs both, including a binary copy of the ecCodes library. Once installed, cfgrib appears in xr.backends.list_engines().

2. Download only the fields you need

GFS files on NOAA's cloud buckets come with an .idx text file listing each message's byte offset, variable, level and step. The 0.25° analysis index listed 696 messages. Requesting byte ranges for just the four messages wanted fetched 3,433,144 bytes instead of 508,053,061 (Example 1). The downloaded pieces concatenate into a valid GRIB2 file.

3. See what the file contains

cfgrib.open_datasets groups messages into every dataset they can form and returns a list: for the 1° analysis file, 35 datasets and 139 variables in 0.712 seconds. Printing each dataset's variables and level coordinate is the quickest map of a GRIB file you have not seen before (Example 2).

4. Filter by level type — and by level when heights differ

A dataset needs one coordinate per dimension. 2 m temperature, 10 m wind, 80 m and 100 m temperature and 1000 m reflectivity are all heightAboveGround, but at different heights, so they cannot share a heightAboveGround coordinate. Filtering on the level type alone made cfgrib keep the first compatible group — reflectivity at 1000 and 4000 m — and skip t2m, sh2, d2m, r2, aptmp, u10, v10, u, v, t, q and pres, reporting only in its log. Add level (2 or 10) or shortName.

5. Filter by step type in forecast files

A forecast step file holds instantaneous fields (temperature, wind), averages over the step (precipitation rate, radiation) and accumulations (total precipitation). Opening the 6-hour file's surface fields raised DatasetBuildError and suggested three filters: stepType instant, avg and accum. With stepType="accum" the file gave tp, acpcp and watr; with avg, prate and 17 others.

6. Convert units and meaning

prate is in kg m⁻² s⁻¹: its unweighted grid mean of 2.83 × 10⁻⁵ is 2.44 mm per day. tp is in kg m⁻², a depth in millimetres accumulated over the step's range, here 0 to 6 hours. GFS accumulations reset at intervals, so differencing consecutive steps needs the step range from the message, not an assumption.

7. Combine forecast steps along step

Files for successive steps open together with open_mfdataset(..., combine="nested", concat_dim="step"). The 2 m fields from the 0, 3 and 6-hour files formed one dataset with a step dimension of 3 and a valid_time coordinate from 00 to 06 UTC in 1.34 seconds.

8. Mind longitude, latitude order and index files

GFS longitudes run from 0 to 359.75 and latitudes from 90 down to −90, so London sits at 359.75° E (or 0.0°) and latitude slices run high to low. By default cfgrib writes an .idx index next to the GRIB file on first open: the first filtered open of the 1° file took 0.614 s and wrote a 65 kB index, the second 0.005 s, and an open with indexpath="" 0.473 s. In a read-only data directory or a shared bucket, set indexpath to an empty string or a writable location.

Bar chart comparing the full 0.25 degree GFS analysis file size with the four messages downloaded by byte range.
The index file turns a half-gigabyte download into a few megabytes.

Code examples

Example 1 — download selected messages by byte range 

import requests

URL = "https://noaa-gfs-bdp-pds.s3.amazonaws.com/gfs.20260910/00/atmos/gfs.t00z.pgrb2.0p25.f000"


def grib_ranges(idx_text, wanted):
    lines = idx_text.splitlines()
    starts = [int(line.split(":")[1]) for line in lines]
    for i, line in enumerate(lines):
        if any(w in line for w in wanted):
            end = starts[i + 1] - 1 if i + 1 < len(lines) else ""
            yield line, starts[i], end


def download_messages(url, wanted, path):
    idx = requests.get(url + ".idx", timeout=60).text
    total = 0
    with open(path, "wb") as out:
        for line, start, end in grib_ranges(idx, wanted):
            chunk = requests.get(url, headers={"Range": f"bytes={start}-{end}"}, timeout=60).content
            out.write(chunk)
            total += len(chunk)
            print(f"{line.split(':', 3)[3]:40} {len(chunk):>9,} bytes")
    return total


wanted = [":TMP:2 m above ground:", ":UGRD:10 m above ground:", ":VGRD:10 m above ground:", ":PRMSL:mean sea level:"]
print(f"{download_messages(URL, wanted, 'gfs_subset.grib2'):,} bytes")
PRMSL:mean sea level:anl:                  996,547 bytes
TMP:2 m above ground:anl:                  511,469 bytes
UGRD:10 m above ground:anl:                972,878 bytes
VGRD:10 m above ground:anl:                952,250 bytes
3,433,144 bytes

The match strings include the colons on both sides so that TMP:2 m above ground does not also match TMP:2 m above ground at other steps or TMAX.

Example 2 — map a GRIB file, then open one group 

import cfgrib
import xarray as xr

parts = cfgrib.open_datasets("gfs.t00z.pgrb2.1p00.f000", backend_kwargs={"indexpath": ""})
print(f"{len(parts)} datasets, {sum(len(p.data_vars) for p in parts)} variables")
for part in parts:
    levels = [c for c in part.coords if c not in ("time", "step", "valid_time", "latitude", "longitude")]
    if any(v in part.data_vars for v in ("t2m", "u10", "refd")):
        print(f"  {sorted(part.data_vars)} on {levels}")

subset = {"indexpath": ""}
for keys in ({"typeOfLevel": "heightAboveGround", "level": 2},
             {"typeOfLevel": "heightAboveGround", "level": 10},
             {"typeOfLevel": "meanSea"}):
    ds = xr.open_dataset("gfs_subset.grib2", engine="cfgrib", backend_kwargs={"filter_by_keys": keys} | subset)
    for name, var in ds.data_vars.items():
        print(f"{keys}: {name} {var.shape} {var.attrs['GRIB_units']}, grid mean {float(var.mean()):.2f}")
35 datasets, 139 variables
  ['u10', 'v10'] on ['heightAboveGround']
  ['aptmp', 'd2m', 'r2', 'sh2', 't2m'] on ['heightAboveGround']
  ['refd'] on ['heightAboveGround']
  ['refd'] on ['hybrid']
{'typeOfLevel': 'heightAboveGround', 'level': 2}: t2m (721, 1440) K, grid mean 280.10
{'typeOfLevel': 'heightAboveGround', 'level': 10}: u10 (721, 1440) m s**-1, grid mean 0.44
{'typeOfLevel': 'heightAboveGround', 'level': 10}: v10 (721, 1440) m s**-1, grid mean 0.44
{'typeOfLevel': 'meanSea'}: prmsl (721, 1440) Pa, grid mean 100965.67

cfgrib.open_datasets also logs a warning for every message group it could not merge; those lines are omitted here.

Example 3 — forecast steps, step types and units 

def open_grib(path, **keys):
    return xr.open_dataset(path, engine="cfgrib", backend_kwargs={"filter_by_keys": keys, "indexpath": ""})


try:
    open_grib("gfs.t00z.pgrb2.1p00.f006", typeOfLevel="surface")
except Exception as error:
    print(type(error).__name__, "-", str(error).splitlines()[0])

prate = open_grib("gfs.t00z.pgrb2.1p00.f006", typeOfLevel="surface", stepType="avg", shortName="prate")["prate"]
tp = open_grib("gfs.t00z.pgrb2.1p00.f006", typeOfLevel="surface", stepType="accum", shortName="tp")["tp"]
print(f"prate [{prate.attrs['units']}] grid mean {float(prate.mean()) * 86400:.2f} mm/day, valid {str(prate.valid_time.values)[:16]}")
print(f"tp [{tp.attrs['units']}] grid mean {float(tp.mean()):.3f} mm over the step")

steps = xr.open_mfdataset(
    ["gfs.t00z.pgrb2.1p00.f000", "gfs.t00z.pgrb2.1p00.f003", "gfs.t00z.pgrb2.1p00.f006"],
    engine="cfgrib", combine="nested", concat_dim="step",
    backend_kwargs={"filter_by_keys": {"typeOfLevel": "heightAboveGround", "level": 2}, "indexpath": ""},
)
print(dict(steps.sizes), [str(t)[11:16] for t in steps.valid_time.values], sorted(steps.data_vars))
DatasetBuildError - multiple values for unique key, try re-open the file with one of:
prate [kg m**-2 s**-1] grid mean 2.44 mm/day, valid 2026-09-10T06:00
tp [kg m**-2] grid mean 0.609 mm over the step
{'step': 3, 'latitude': 181, 'longitude': 360} ['00:00', '03:00', '06:00'] ['aptmp', 'd2m', 'r2', 'sh2', 't2m', 'tmax', 'tmin']

Explanation

Why a GRIB file is not a dataset

Each GRIB message is self-contained: its own grid description, parameter, level, reference time and step. Nothing requires messages in one file to share levels or steps. xarray's data model needs named dimensions with one coordinate each, so cfgrib must group messages whose coordinates line up, and a file with many level types and step types has many such groups.

Why cfgrib skips variables instead of failing

When a filter still allows incompatible messages, cfgrib builds the dataset from the variables it can combine and logs a skipping variable message for each of the rest. That keeps partial reads working, and it means a filter that is too loose returns fewer variables than expected rather than raising. Check the variable list every time.

Why step type matters for precipitation

A 6-hour GFS file stores precipitation twice: as a rate averaged over the step and as an accumulation. They have different units, different reductions and different relationships to neighbouring steps. stepType is the key that distinguishes them, and GRIB_stepRange records the period.

Why byte ranges work

GRIB2 messages are concatenated with no file header, so any subset of whole messages is itself a valid file. The .idx file records where each message starts; the next message's start marks where it ends.

Two panels comparing a filter on typeOfLevel heightAboveGround alone, which returned only radar reflectivity and skipped twelve variables, with a filter that adds level 2.
A filter that is too loose does not fail; it returns fewer variables.

Edge cases or notes

  • ECMWF open data uses the same keys; its files also mix level and step types.
  • Ensemble members add a number dimension; filter or keep it.
  • Rotated or Lambert grids decode with 2-D latitude and longitude coordinates.
  • GRIB1 and GRIB2 both work through ecCodes; parameter names can differ between centres.
  • Index files in shared directories can go stale when the GRIB file is replaced; delete them or disable indexing.
  • Large archives are often better converted once to NetCDF or Zarr; see GRIB, NetCDF or Zarr.
  • Forecast reference and valid time are different coordinates: time is when the forecast started, valid_time when it applies.

FAQ

How do I open a GRIB2 file in Python?

Install cfgrib and eccodes, then call xr.open_dataset(path, engine="cfgrib") with backend_kwargs={"filter_by_keys": {...}} selecting one typeOfLevel, plus level or shortName and stepType as needed.

What does "DatasetBuildError: multiple values for unique key" mean?

The file holds messages that cannot share coordinates, such as several level types or step types. Filter with filter_by_keys, or use cfgrib.open_datasets to get every compatible group.

Why is 2 m temperature missing from my GRIB dataset?

The filter allowed other heightAboveGround levels, so cfgrib kept an incompatible group and skipped t2m. Add "level": 2 to filter_by_keys.

How do I download only some variables from a GFS file?

Read the .idx file, find the byte offsets of the messages you want and request those ranges. Four fields from the 508 MB 0.25° file were 3.43 MB.

What is the difference between prate and tp in GFS?

prate is a precipitation rate in kg m⁻² s⁻¹ averaged over the step; tp is total precipitation in kg m⁻² accumulated over the step range. Multiply prate by 86,400 for mm per day.

Why does cfgrib create .idx files next to my data?

It caches an index of message offsets to speed up later opens. Pass "indexpath": "" in backend_kwargs to disable it, or point it at a writable directory.