GRIB, NetCDF or Zarr: Choosing a Format for Gridded Data

On this page

Problem statement

The same forecast or reanalysis field can arrive as GRIB2, NetCDF or Zarr, and converting between them is routine. The choice decides how large the files are, how fast a map or a time series comes back, whether the data can be read straight from cloud storage, and how much of the meaning travels with the numbers. None of the three is best at everything.

Measured by writing the same GFS 1° upper-air analysis — temperature, wind components and geopotential height on 33 pressure levels, 34.4 MB as 32-bit floats — in each format:

GRIB2 was the smallest at 9.4 MB, 27.3% of the raw size, and by far the slowest to open: 85 ms against 3–5 ms for NetCDF and Zarr.
Compressed NetCDF was 19.9 MB; with 4 significant digits of quantization, 13.9 MB, at a cost of at most 0.0078 K.
Zarr v3 was 18.4 MB spread over 143 files; Zarr v2 with consolidated metadata 24.8 MB over 152.
Remotely, a 1,980-month CMIP6 Zarr store on Google Cloud Storage opened in 1.85 s; a 1.53 MB NetCDF file read by 16 KB HTTP byte ranges took 8.56 s just to open.

All three lossless versions decoded to identical values.

Quick answer

situation                                           format
receiving operational forecasts                     GRIB2, as published; convert what you keep
archiving and exchanging files with other tools     NetCDF-4 with compression and CF attributes
analysing large arrays in parallel or from S3/GCS   Zarr with sensible chunks and consolidated metadata
shipping to a GIS                                   GeoTIFF / COG, one variable and time at a time

Read GRIB with cfgrib, then write whatever you will work with:

import xarray as xr

ds = xr.open_dataset("upper.grib2", engine="cfgrib",
                     backend_kwargs={"filter_by_keys": {"typeOfLevel": "isobaricInhPa"}, "indexpath": ""})
ds = ds.drop_vars(["time", "step", "valid_time"])
ds.to_netcdf("upper.nc", encoding={v: {"zlib": True, "complevel": 4, "shuffle": True} for v in ds.data_vars})
ds.chunk({"isobaricInhPa": 1}).to_zarr("upper.zarr", mode="w", consolidated=True)

GRIB's packing beat general-purpose compression; quantization closed most of the gap.

What each format is, then size, speed and chunks

1. Know what each format is

GRIB2 is a sequence of independent messages, one field each, designed by the World Meteorological Organization for transmitting forecasts compactly. NetCDF-4 is a single self-describing file of named arrays and attributes stored in HDF5. Zarr stores each array as a directory of separately compressed chunks plus small JSON metadata files, designed for parallel and cloud access. See NetCDF and gridded data explained and chunked arrays and Zarr explained.

2. Compare size on your own data

GRIB2 packs each field as integers with a per-field scale, using complex spatial differencing: here 9 to 15 bits per value depending on the variable and level, 9,379,220 bytes for the 132 messages. Plain NetCDF stored the 32-bit floats as they were, 34,446,538 bytes. zlib level 4 with the shuffle filter reduced that to 19,936,493; adding significant_digits=4 quantization to 13,871,820. Zarr v3 with its default zstd codec took 18,431,781 bytes; Zarr v2 with consolidated metadata, 24,809,736 in 152 files (Example 4).

3. Compare read speed for the reads you will do

format                 open      one 500 hPa field   all temperature levels
GRIB (cfgrib)         85.0 ms         85.3 ms              119.4 ms
NetCDF, zlib           2.9 ms          3.5 ms               24.4 ms
NetCDF, uncompressed   2.9 ms          2.8 ms                3.6 ms
Zarr v3                4.5 ms          6.8 ms               22.9 ms
Zarr v2, consolidated  3.6 ms          5.6 ms               20.7 ms

Opening GRIB means scanning messages to build the dataset, every time unless an index file is kept. Compressed formats trade decompression time for size; uncompressed NetCDF was fastest to read and 3.7 times the size of GRIB. The times are the best of five runs (three for all levels) from Examples 2 and 4, on a machine shared with other work; single milliseconds move between runs, the gap to GRIB does not.

4. Match chunks to access patterns

NetCDF-4 and Zarr both split arrays into chunks, and a read decompresses every chunk it touches. The files above were chunked one pressure level per chunk, which makes single-level maps cheap and full vertical profiles expensive. GRIB has no chunking beyond one message per field. Choose chunks from the reads you expect — see choosing chunk and tile sizes.

Zarr keeps its metadata where one request finds it; HDF5 makes a remote reader go looking.

5. Decide where the data will live

On local disk all three are practical. From object storage the difference is large. Zarr's metadata is a few small objects and each chunk is one request: a CMIP6 monthly temperature store on Google Cloud Storage, 1,980 months of a 180 × 288 grid, opened in 1.85 s and returned its first month in 2.90 s. A NetCDF file read over HTTP needs many small range requests to find HDF5 metadata scattered through the file: one 1.53 MB daily OISST file took 8.56 s to open with 16 KB reads and 0.92 s more to read a 10° box. Both depend on the network — an earlier run took 2.75 s and 12.90 s to open the two — so the point is how the time scales with the number of requests, not the seconds. GRIB over HTTP is practical only with an index of byte ranges, as in reading GRIB weather data.

6. Consider what metadata travels with the data

NetCDF and Zarr carry arbitrary attributes, and CF conventions give them shared meaning: units, standard names, bounds, calendars. GRIB identifies parameters by numeric codes in WMO and centre-specific tables; cfgrib translates them to names and CF-like attributes, but local parameters and less common levels can decode as unknown. When converting from GRIB, check variable names and units and add what is missing — see CF conventions explained.

7. Convert once, not on every read

Reading GRIB repeatedly pays the scan cost repeatedly and leaves the message-grouping problems of cfgrib in every script. For data you will analyse more than once, convert the variables you need to NetCDF or Zarr in one step, with explicit encoding, and keep the GRIB as the archive of record.

8. Verify that the conversion was lossless — or how lossy

Decode the source and the converted copy and compare. Compressed NetCDF and Zarr matched the GRIB decode exactly, a maximum absolute difference of 0.0. The quantized NetCDF differed by at most 0.0078 K, which is smaller than GRIB's own packing step of 0.01 K for every temperature field (Example 4) but is still a deliberate loss (Example 3).

GRIB's cost is in opening: cfgrib rebuilds the dataset from messages every time.

One GFS field set in three formats

The data used on this page

NOAA's GFS analysis from 10 September 2026, 00 UTC, from the NOAA Open Data Dissemination bucket: the 1° file gfs.t00z.pgrb2.1p00.f000 and its .idx index, from which Example 1 copies the temperature, wind and height messages on pressure levels. Example 4 also reads two public remote datasets: a CMIP6 near-surface temperature store on Google Cloud Storage and one daily NOAA OISST file from NCEI.

import os
import urllib.request

BASE = "https://noaa-gfs-bdp-pds.s3.amazonaws.com/gfs.20260910/00/atmos/"
for name in ("gfs.t00z.pgrb2.1p00.f000", "gfs.t00z.pgrb2.1p00.f000.idx"):
    if not os.path.exists(name):
        urllib.request.urlretrieve(BASE + name, name)
print(f"{os.path.getsize('gfs.t00z.pgrb2.1p00.f000') / 1e6:.1f} MB, "
      f"{len(open('gfs.t00z.pgrb2.1p00.f000.idx').read().splitlines())} messages")

42.4 MB, 696 messages

Example 1 — extract the same field set into three formats

import os
import shutil

import xarray as xr


def extract_messages(grib_path, idx_path, keep, out_path):
    """Copy whole GRIB messages whose index line matches keep(line) into a new file."""
    lines = open(idx_path).read().splitlines()
    starts = [int(line.split(":")[1]) + 0 for line in lines] + [os.path.getsize(grib_path)]
    with open(grib_path, "rb") as src, open(out_path, "wb") as out:
        for i, line in enumerate(lines):
            if keep(line):
                src.seek(starts[i])
                out.write(src.read(starts[i + 1] - starts[i]))


def size_on_disk(path):
    if os.path.isfile(path):
        return os.path.getsize(path), 1
    files = [os.path.join(d, f) for d, _, names in os.walk(path) for f in names]
    return sum(os.path.getsize(f) for f in files), len(files)


def upper_air(line):
    """TMP/UGRD/VGRD/HGT on the 1-1000 hPa levels; the 0.01-0.7 mb ones decode as a separate Pa level type."""
    var, level = line.split(":")[3:5]
    return var in ("TMP", "UGRD", "VGRD", "HGT") and level.endswith(" mb") and float(level.split()[0]) >= 1


extract_messages("gfs.t00z.pgrb2.1p00.f000", "gfs.t00z.pgrb2.1p00.f000.idx", upper_air, "upper.grib2")
ds = xr.open_dataset("upper.grib2", engine="cfgrib",
                     backend_kwargs={"filter_by_keys": {"typeOfLevel": "isobaricInhPa"}, "indexpath": ""})
ds = ds.drop_vars(["time", "step", "valid_time"]).load()
for name in ds.data_vars:
    ds[name].encoding = {}
chunks = {v: (1, ds.sizes["latitude"], ds.sizes["longitude"]) for v in ds.data_vars}
raw_bytes = sum(ds[v].nbytes for v in ds.data_vars)

for path in ("upper.nc", "upper_zlib.nc", "upper_quantized.nc", "upper.zarr"):
    shutil.rmtree(path, ignore_errors=True) if os.path.isdir(path) else None
ds.to_netcdf("upper.nc")
ds.to_netcdf("upper_zlib.nc", encoding={v: dict(zlib=True, complevel=4, shuffle=True, chunksizes=chunks[v]) for v in ds.data_vars})
ds.to_netcdf("upper_quantized.nc", encoding={v: dict(zlib=True, complevel=4, shuffle=True, chunksizes=chunks[v],
                                                     significant_digits=4) for v in ds.data_vars})
ds.chunk({"isobaricInhPa": 1}).to_zarr("upper.zarr", mode="w", zarr_format=3, consolidated=False)

for path in ("upper.grib2", "upper.nc", "upper_zlib.nc", "upper_quantized.nc", "upper.zarr"):
    size, files = size_on_disk(path)
    print(f"{path:20} {size:>11,} bytes  {size / raw_bytes:6.1%} of float32  {files} file(s)")

upper.grib2            9,379,220 bytes   27.3% of float32  1 file(s)
upper.nc              34,446,538 bytes  100.1% of float32  1 file(s)
upper_zlib.nc         19,936,493 bytes   57.9% of float32  1 file(s)
upper_quantized.nc    13,871,820 bytes   40.3% of float32  1 file(s)
upper.zarr            18,431,781 bytes   53.6% of float32  143 file(s)

Example 2 — time the same reads in each format

import time


def fastest(fn, repeats=5):
    best = float("inf")
    for _ in range(repeats):
        start = time.perf_counter()
        fn()
        best = min(best, time.perf_counter() - start)
    return best * 1000


OPENERS = {
    "GRIB (cfgrib)": lambda: xr.open_dataset("upper.grib2", engine="cfgrib", backend_kwargs={
        "filter_by_keys": {"typeOfLevel": "isobaricInhPa", "shortName": "t"}, "indexpath": ""}),
    "NetCDF, zlib": lambda: xr.open_dataset("upper_zlib.nc"),
    "NetCDF, uncompressed": lambda: xr.open_dataset("upper.nc"),
    "Zarr v3": lambda: xr.open_zarr("upper.zarr", consolidated=False),
}
for name, open_ in OPENERS.items():
    t_open = fastest(open_)
    t_level = fastest(lambda: open_()["t"].sel(isobaricInhPa=500).values)
    t_all = fastest(lambda: open_()["t"].values, repeats=3)
    print(f"{name:22} open {t_open:6.1f} ms   500 hPa field {t_level:6.1f} ms   all levels {t_all:6.1f} ms")

GRIB (cfgrib)          open   85.0 ms   500 hPa field   85.3 ms   all levels  119.4 ms
NetCDF, zlib           open    2.9 ms   500 hPa field    3.5 ms   all levels   24.4 ms
NetCDF, uncompressed   open    2.9 ms   500 hPa field    2.8 ms   all levels    3.6 ms
Zarr v3                open    4.5 ms   500 hPa field    6.8 ms   all levels   22.9 ms

These are the rows of the step 3 table; Example 4 adds Zarr v2.

GRIB is already rounded to the producer’s step; the quantization error is smaller than that rounding, though still a change to the decoded values.

Example 3 — check that the copies decode to the same values

import numpy as np

source = OPENERS["GRIB (cfgrib)"]()["t"].values
for name in ("NetCDF, zlib", "Zarr v3"):
    print(f"{name:14} max abs difference {float(np.nanmax(np.abs(OPENERS[name]()['t'].values - source))):.4f} K")
quantized = xr.open_dataset("upper_quantized.nc")["t"].values
print(f"{'quantized':14} max abs difference {float(np.nanmax(np.abs(quantized - source))):.4f} K")

NetCDF, zlib   max abs difference 0.0000 K
Zarr v3        max abs difference 0.0000 K
quantized      max abs difference 0.0078 K

Example 4 — Zarr v2, GRIB packing and remote reads

import eccodes
import fsspec

shutil.rmtree("upper_v2.zarr", ignore_errors=True)
ds.chunk({"isobaricInhPa": 1}).to_zarr("upper_v2.zarr", mode="w", zarr_format=2, consolidated=True)
size, files = size_on_disk("upper_v2.zarr")
print(f"Zarr v2, consolidated: {size:,} bytes ({size / 1e6:.1f} MB) in {files} files")
v2 = lambda: xr.open_zarr("upper_v2.zarr", consolidated=True)
print(f"  open {fastest(v2):.1f} ms   500 hPa field {fastest(lambda: v2()['t'].sel(isobaricInhPa=500).values):.1f} ms"
      f"   all levels {fastest(lambda: v2()['t'].values, repeats=3):.1f} ms")

bits, step = {}, {}
with open("upper.grib2", "rb") as f:
    while (h := eccodes.codes_grib_new_from_file(f)) is not None:
        name = eccodes.codes_get(h, "shortName")
        bits.setdefault(name, []).append(eccodes.codes_get(h, "bitsPerValue"))
        step.setdefault(name, []).append(2.0 ** eccodes.codes_get(h, "binaryScaleFactor")
                                         * 10.0 ** -eccodes.codes_get(h, "decimalScaleFactor"))
        packing = eccodes.codes_get(h, "packingType")
        eccodes.codes_release(h)
print(f"GRIB, {sum(map(len, bits.values()))} messages, {packing}")
for name in bits:
    print(f"  {name:3} bits per value {min(bits[name])}-{max(bits[name])}   "
          f"packing step {min(step[name]):g}-{max(step[name]):g}")

CMIP6 = "gs://cmip6/CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/historical/r1i1p1f1/Amon/tas/gr1/v20190726/"
start = time.perf_counter()
cm = xr.open_zarr(fsspec.get_mapper(CMIP6, token="anon"), consolidated=True)
t_open = time.perf_counter() - start
start = time.perf_counter()
cm["tas"].isel(time=0).values
print(f"CMIP6 Zarr on GCS {cm['tas'].shape}: open {t_open:.2f} s, first month {time.perf_counter() - start:.2f} s")

OISST = ("https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/"
         "v2.1/access/avhrr/202409/oisst-avhrr-v02r01.20240901.nc")
fs = fsspec.filesystem("https")
print(f"OISST NetCDF: {fs.size(OISST) / 1e6:.2f} MB")
start = time.perf_counter()
remote = xr.open_dataset(fs.open(OISST, block_size=2**14, cache_type="none"), engine="h5netcdf")
t_open = time.perf_counter() - start
start = time.perf_counter()
remote["sst"].sel(lat=slice(40, 50), lon=slice(0, 10)).values
print(f"  over HTTP ranges: open {t_open:.2f} s, 10° box {time.perf_counter() - start:.2f} s")

Zarr v2, consolidated: 24,809,736 bytes (24.8 MB) in 152 files
  open 3.6 ms   500 hPa field 5.6 ms   all levels 20.7 ms
GRIB, 132 messages, grid_complex_spatial_differencing
  gh  bits per value 11-15   packing step 0.016-0.16
  t   bits per value 11-13   packing step 0.01-0.01
  u   bits per value 9-14   packing step 0.01-0.1
  v   bits per value 10-13   packing step 0.01-0.1
CMIP6 Zarr on GCS (1980, 180, 288): open 1.85 s, first month 2.90 s
OISST NetCDF: 1.53 MB
  over HTTP ranges: open 8.56 s, 10° box 0.92 s

Why GRIB is small and slow, and Zarr suits object storage

Why GRIB is small

GRIB2 does not store floats. Each field is scaled to integers with a precision chosen by the producer, and complex packing with spatial differencing stores the differences between neighbouring values in as few bits as each group needs. It is lossy at the producer's chosen precision, but that loss happened before you received the file; a lossless copy in another format preserves exactly the decoded values.

Why GRIB is slow to open

A GRIB file has no central directory of variables and dimensions. cfgrib reads the header of every message, groups compatible messages, and builds coordinates, which is work proportional to the number of messages. An index file speeds up later opens of the same file; a converted NetCDF or Zarr copy removes the work entirely.

Why Zarr suits object storage

Object stores charge latency per request and have no cheap way to read scattered small pieces of one large object. Zarr puts metadata in a few small objects and each chunk in its own object, so opening is one or two requests and reading is one request per chunk, easily run in parallel. The price is many files on a local disk: 143 for this small store.

Why quantization is worth considering

Most model and reanalysis values carry more digits than their accuracy supports. Rounding to four significant digits before compression let zlib find far more repetition, cutting the NetCDF from 19.9 to 13.9 MB with a maximum error below one hundredth of a kelvin. Whether that is acceptable depends on the variable and the use, and it should be recorded in the file's attributes.

The format follows the use: receive in GRIB, exchange in NetCDF, compute at scale in Zarr, hand to GIS as GeoTIFF.

NetCDF-3, Zarr versions and virtual references

NetCDF-3 has no compression or chunking; files converted to it are always the size of the raw arrays.
Zarr v2 and v3 differ in metadata layout and default codecs; not every tool reads v3 yet.
Consolidated metadata in Zarr v2 saves one request per variable on open; it must be refreshed after appending.
Kerchunk and VirtualiZarr can index existing NetCDF or GRIB files so they read like Zarr without conversion.
Many small Zarr chunks are slow on local file systems with high per-file overhead.
GRIB parameter tables differ by centre; the same shortName can mean different things in ECMWF and NCEP files.
Timings depend on hardware and load; measure on your own storage before committing an archive to a format.

Internal links

How to read GRIB weather forecast data in Python — getting data out of GRIB
NetCDF and gridded data explained: dimensions, variables and attributes — the NetCDF model
Chunked arrays and Zarr explained — how Zarr stores arrays
How to choose chunk and tile sizes that actually help — chunking for access patterns
Cloud-optimised GeoTIFF explained: tiles, overviews and range requests — the raster equivalent for the cloud
How to open hundreds of NetCDF files as one dataset — when a collection outgrows single files
CF conventions explained: how a NetCDF file says what its numbers mean — metadata after conversion
How to convert NetCDF to GeoTIFF in Python — the GIS hand-off

FAQ

Is GRIB or NetCDF smaller?

GRIB2 usually. The GFS upper-air subset was 9.4 MB as GRIB2, 19.9 MB as compressed NetCDF and 13.9 MB as quantized, compressed NetCDF.

Should I convert GRIB to NetCDF?

For data you will analyse repeatedly, yes. Opening the GRIB took 85 ms and required cfgrib's message grouping each time; the compressed NetCDF opened in 2.9 ms with identical values.

When is Zarr better than NetCDF?

When reading from cloud object storage or computing in parallel over many chunks. A 1,980-month CMIP6 Zarr store opened from Google Cloud Storage in 1.85 s, while a 1.53 MB NetCDF file over HTTP range requests took 8.56 s.

Does converting GRIB to NetCDF or Zarr lose precision?

Not with lossless compression: both copies matched the GRIB decode exactly. Quantization is a deliberate loss; four significant digits changed temperatures by at most 0.0078 K.

Why does my Zarr store have so many files?

Each chunk of each variable is a separate object. The small GFS store had 143 files; that suits object storage and can be slow on some local file systems.

Can I read NetCDF directly from S3 or HTTPS?

Yes, with byte-range access, but HDF5 metadata makes it slow. Kerchunk-style references or converting to Zarr give much faster remote reads.