GRIB, NetCDF or Zarr: Choosing a Format for Gridded Data
Problem statement
The same forecast or reanalysis field can arrive as GRIB2, NetCDF or Zarr, and converting between them is routine. The choice decides how large the files are, how fast a map or a time series comes back, whether the data can be read straight from cloud storage, and how much of the meaning travels with the numbers. None of the three is best at everything.
Measured by writing the same GFS 1° upper-air analysis — temperature, wind components and geopotential height on 33 pressure levels, 34.4 MB as 32-bit floats — in each format:
- GRIB2 was the smallest at 11.4 MB, 33.1% of the raw size, and by far the slowest to open: 115 ms against 3–6 ms for NetCDF and Zarr.
- Compressed NetCDF was 19.9 MB; with 4 significant digits of quantization, 13.9 MB, at a cost of at most 0.0078 K.
- Zarr v3 was 18.4 MB spread over 143 files; Zarr v2 with consolidated metadata 24.8 MB over 152.
- From cloud storage, a Zarr store opened in 0.80 s; a 1.5 MB NetCDF file read by HTTP byte ranges took 9.11 s to open and another 5.36 s to read a small box.
All three lossless versions decoded to identical values.
Quick answer
situation format
receiving operational forecasts GRIB2, as published; convert what you keep
archiving and exchanging files with other tools NetCDF-4 with compression and CF attributes
analysing large arrays in parallel or from S3/GCS Zarr with sensible chunks and consolidated metadata
shipping to a GIS GeoTIFF / COG, one variable and time at a time
Read GRIB with cfgrib, then write whatever you will work with:
import xarray as xr
ds = xr.open_dataset("upper.grib2", engine="cfgrib",
backend_kwargs={"filter_by_keys": {"typeOfLevel": "isobaricInhPa"}, "indexpath": ""})
ds = ds.drop_vars(["time", "step", "valid_time"])
ds.to_netcdf("upper.nc", encoding={v: {"zlib": True, "complevel": 4, "shuffle": True} for v in ds.data_vars})
ds.chunk({"isobaricInhPa": 1}).to_zarr("upper.zarr", mode="w", consolidated=True)
Step-by-step solution
1. Know what each format is
GRIB2 is a sequence of independent messages, one field each, designed by the World Meteorological Organization for transmitting forecasts compactly. NetCDF-4 is a single self-describing file of named arrays and attributes stored in HDF5. Zarr stores each array as a directory of separately compressed chunks plus small JSON metadata files, designed for parallel and cloud access. See NetCDF and gridded data explained and chunked arrays and Zarr explained.
2. Compare size on your own data
GRIB2 packs each field as integers with a per-field scale, using complex spatial differencing: here 9 to 15 bits per value depending on the variable and level, 11,400,506 bytes in all. Plain NetCDF stored the 32-bit floats as they were, 34,446,538 bytes. zlib level 4 with the shuffle filter reduced that to 19,936,493; adding significant_digits=4 quantization to 13,871,820. Zarr v3 with its default zstd codec took 18,431,781 bytes; Zarr v2 with consolidated metadata, in an earlier run, 24,809,780.
3. Compare read speed for the reads you will do
format open one 500 hPa field all temperature levels
GRIB (cfgrib) 115.1 ms 116.6 ms 158.0 ms
NetCDF, zlib 2.8 ms 4.5 ms 29.5 ms
NetCDF, uncompressed 3.2 ms 2.9 ms 5.9 ms
Zarr v3 5.5 ms 7.1 ms 26.3 ms
Zarr v2, consolidated 4.0 ms 6.2 ms 29.0 ms
Opening GRIB means scanning messages to build the dataset, every time unless an index file is kept. Compressed formats trade decompression time for size; uncompressed NetCDF was fastest to read and three times the size of GRIB.
4. Match chunks to access patterns
NetCDF-4 and Zarr both split arrays into chunks, and a read decompresses every chunk it touches. The files above were chunked one pressure level per chunk, which makes single-level maps cheap and full vertical profiles expensive. GRIB has no chunking beyond one message per field. Choose chunks from the reads you expect — see choosing chunk and tile sizes.
5. Decide where the data will live
On local disk all three are practical. From object storage the difference is large. Zarr's metadata is a few small objects and each chunk is one request: a CMIP6 temperature store on Google Cloud Storage opened in 0.80 s and returned its first month in 2.68 s. A NetCDF file read over HTTP needs many small range requests to find HDF5 metadata scattered through the file: the 1.5 MB OISST file took 9.11 s to open, and a 29.6 MB reanalysis file took 185.8 s. GRIB over HTTP is practical only with an index of byte ranges, as in reading GRIB weather data.
6. Consider what metadata travels with the data
NetCDF and Zarr carry arbitrary attributes, and CF conventions give them shared meaning: units, standard names, bounds, calendars. GRIB identifies parameters by numeric codes in WMO and centre-specific tables; cfgrib translates them to names and CF-like attributes, but local parameters and less common levels can decode as unknown. When converting from GRIB, check variable names and units and add what is missing — see CF conventions explained.
7. Convert once, not on every read
Reading GRIB repeatedly pays the scan cost repeatedly and leaves the message-grouping problems of cfgrib in every script. For data you will analyse more than once, convert the variables you need to NetCDF or Zarr in one step, with explicit encoding, and keep the GRIB as the archive of record.
8. Verify that the conversion was lossless — or how lossy
Decode the source and the converted copy and compare. Compressed NetCDF and Zarr matched the GRIB decode exactly, a maximum absolute difference of 0.0. The quantized NetCDF differed by at most 0.0078 K, which is smaller than GRIB's own packing step for most fields but is still a deliberate loss (Example 3).
Code examples
Example 1 — extract the same field set into three formats
import os
import shutil
import xarray as xr
def extract_messages(grib_path, idx_path, keep, out_path):
"""Copy whole GRIB messages whose index line matches keep(line) into a new file."""
lines = open(idx_path).read().splitlines()
starts = [int(line.split(":")[1]) + 0 for line in lines] + [os.path.getsize(grib_path)]
with open(grib_path, "rb") as src, open(out_path, "wb") as out:
for i, line in enumerate(lines):
if keep(line):
src.seek(starts[i])
out.write(src.read(starts[i + 1] - starts[i]))
def size_on_disk(path):
if os.path.isfile(path):
return os.path.getsize(path), 1
files = [os.path.join(d, f) for d, _, names in os.walk(path) for f in names]
return sum(os.path.getsize(f) for f in files), len(files)
extract_messages("gfs.t00z.pgrb2.1p00.f000", "gfs.t00z.pgrb2.1p00.f000.idx",
lambda line: line.split(":")[3] in ("TMP", "UGRD", "VGRD", "HGT") and line.split(":")[4].endswith(" mb"),
"upper.grib2")
ds = xr.open_dataset("upper.grib2", engine="cfgrib",
backend_kwargs={"filter_by_keys": {"typeOfLevel": "isobaricInhPa"}, "indexpath": ""})
ds = ds.drop_vars(["time", "step", "valid_time"]).load()
for name in ds.data_vars:
ds[name].encoding = {}
chunks = {v: (1, ds.sizes["latitude"], ds.sizes["longitude"]) for v in ds.data_vars}
raw_bytes = sum(ds[v].nbytes for v in ds.data_vars)
for path in ("upper.nc", "upper_zlib.nc", "upper_quantized.nc", "upper.zarr"):
shutil.rmtree(path, ignore_errors=True) if os.path.isdir(path) else None
ds.to_netcdf("upper.nc")
ds.to_netcdf("upper_zlib.nc", encoding={v: dict(zlib=True, complevel=4, shuffle=True, chunksizes=chunks[v]) for v in ds.data_vars})
ds.to_netcdf("upper_quantized.nc", encoding={v: dict(zlib=True, complevel=4, shuffle=True, chunksizes=chunks[v],
significant_digits=4) for v in ds.data_vars})
ds.chunk({"isobaricInhPa": 1}).to_zarr("upper.zarr", mode="w", zarr_format=3, consolidated=False)
for path in ("upper.grib2", "upper.nc", "upper_zlib.nc", "upper_quantized.nc", "upper.zarr"):
size, files = size_on_disk(path)
print(f"{path:20} {size:>11,} bytes {size / raw_bytes:6.1%} of float32 {files} file(s)")upper.grib2 11,400,506 bytes 33.1% of float32 1 file(s)
upper.nc 34,446,538 bytes 100.1% of float32 1 file(s)
upper_zlib.nc 19,936,493 bytes 57.9% of float32 1 file(s)
upper_quantized.nc 13,871,820 bytes 40.3% of float32 1 file(s)
upper.zarr 18,431,781 bytes 53.6% of float32 143 file(s)
Example 2 — time the same reads in each format
import time
def fastest(fn, repeats=5):
best = float("inf")
for _ in range(repeats):
start = time.perf_counter()
fn()
best = min(best, time.perf_counter() - start)
return best * 1000
OPENERS = {
"GRIB (cfgrib)": lambda: xr.open_dataset("upper.grib2", engine="cfgrib", backend_kwargs={
"filter_by_keys": {"typeOfLevel": "isobaricInhPa", "shortName": "t"}, "indexpath": ""}),
"NetCDF, zlib": lambda: xr.open_dataset("upper_zlib.nc"),
"NetCDF, uncompressed": lambda: xr.open_dataset("upper.nc"),
"Zarr v3": lambda: xr.open_zarr("upper.zarr", consolidated=False),
}
for name, open_ in OPENERS.items():
t_open = fastest(open_)
t_level = fastest(lambda: open_()["t"].sel(isobaricInhPa=500).values)
t_all = fastest(lambda: open_()["t"].values, repeats=3)
print(f"{name:22} open {t_open:6.1f} ms 500 hPa field {t_level:6.1f} ms all levels {t_all:6.1f} ms")GRIB (cfgrib) open 115.7 ms 500 hPa field 114.3 ms all levels 152.4 ms
NetCDF, zlib open 3.3 ms 500 hPa field 5.0 ms all levels 28.9 ms
NetCDF, uncompressed open 5.3 ms 500 hPa field 5.6 ms all levels 9.3 ms
Zarr v3 open 6.3 ms 500 hPa field 9.1 ms all levels 42.4 ms
This run was slower for the uncompressed NetCDF and Zarr than the one in step 3; the ranking of GRIB against the rest did not change.
Example 3 — check that the copies decode to the same values
import numpy as np
source = OPENERS["GRIB (cfgrib)"]()["t"].values
for name in ("NetCDF, zlib", "Zarr v3"):
print(f"{name:14} max abs difference {float(np.nanmax(np.abs(OPENERS[name]()['t'].values - source))):.4f} K")
quantized = xr.open_dataset("upper_quantized.nc")["t"].values
print(f"{'quantized':14} max abs difference {float(np.nanmax(np.abs(quantized - source))):.4f} K")NetCDF, zlib max abs difference 0.0000 K
Zarr v3 max abs difference 0.0000 K
quantized max abs difference 0.0078 K
Explanation
Why GRIB is small
GRIB2 does not store floats. Each field is scaled to integers with a precision chosen by the producer, and complex packing with spatial differencing stores the differences between neighbouring values in as few bits as each group needs. It is lossy at the producer's chosen precision, but that loss happened before you received the file; a lossless copy in another format preserves exactly the decoded values.
Why GRIB is slow to open
A GRIB file has no central directory of variables and dimensions. cfgrib reads the header of every message, groups compatible messages, and builds coordinates, which is work proportional to the number of messages. An index file speeds up later opens of the same file; a converted NetCDF or Zarr copy removes the work entirely.
Why Zarr suits object storage
Object stores charge latency per request and have no cheap way to read scattered small pieces of one large object. Zarr puts metadata in a few small objects and each chunk in its own object, so opening is one or two requests and reading is one request per chunk, easily run in parallel. The price is many files on a local disk: 143 for this small store.
Why quantization is worth considering
Most model and reanalysis values carry more digits than their accuracy supports. Rounding to four significant digits before compression let zlib find far more repetition, cutting the NetCDF from 19.9 to 13.9 MB with a maximum error below one hundredth of a kelvin. Whether that is acceptable depends on the variable and the use, and it should be recorded in the file's attributes.
Edge cases or notes
- NetCDF-3 has no compression or chunking; files converted to it are always the size of the raw arrays.
- Zarr v2 and v3 differ in metadata layout and default codecs; not every tool reads v3 yet.
- Consolidated metadata in Zarr v2 saves one request per variable on open; it must be refreshed after appending.
- Kerchunk and VirtualiZarr can index existing NetCDF or GRIB files so they read like Zarr without conversion.
- Many small Zarr chunks are slow on local file systems with high per-file overhead.
- GRIB parameter tables differ by centre; the same
shortNamecan mean different things in ECMWF and NCEP files. - Timings depend on hardware and load; measure on your own storage before committing an archive to a format.
Internal links
- How to read GRIB weather forecast data in Python — getting data out of GRIB
- NetCDF and gridded data explained: dimensions, variables and attributes — the NetCDF model
- Chunked arrays and Zarr explained — how Zarr stores arrays
- How to choose chunk and tile sizes that actually help — chunking for access patterns
- Cloud-optimised GeoTIFF explained: tiles, overviews and range requests — the raster equivalent for the cloud
- How to open hundreds of NetCDF files as one dataset — when a collection outgrows single files
- CF conventions explained: how a NetCDF file says what its numbers mean — metadata after conversion
- How to convert NetCDF to GeoTIFF in Python — the GIS hand-off
FAQ
Is GRIB or NetCDF smaller?
GRIB2 usually. The GFS upper-air subset was 11.4 MB as GRIB2, 19.9 MB as compressed NetCDF and 13.9 MB as quantized, compressed NetCDF.
Should I convert GRIB to NetCDF?
For data you will analyse repeatedly, yes. Opening the GRIB took 115 ms and required cfgrib's message grouping each time; the compressed NetCDF opened in 2.8 ms with identical values.
When is Zarr better than NetCDF?
When reading from cloud object storage or computing in parallel over many chunks. A CMIP6 Zarr store opened from Google Cloud Storage in 0.80 s, while a small NetCDF file over HTTP took 9.11 s.
Does converting GRIB to NetCDF or Zarr lose precision?
Not with lossless compression: both copies matched the GRIB decode exactly. Quantization is a deliberate loss; four significant digits changed temperatures by at most 0.0078 K.
Why does my Zarr store have so many files?
Each chunk of each variable is a separate object. The small GFS store had 143 files; that suits object storage and can be slow on some local file systems.
Can I read NetCDF directly from S3 or HTTPS?
Yes, with byte-range access, but HDF5 metadata makes it slow. Kerchunk-style references or converting to Zarr give much faster remote reads.