⚙️ How-To Gridded & climate data Files & formats Getting started Gridded & Climate Data · step 3 of 20 Updated

How to Open a NetCDF File in Python with xarray

Problem statement

xr.open_dataset("file.nc") usually just works, which hides the decisions it makes: which library reads the file, whether anything is loaded into memory, whether dask is involved, how packed values and times are decoded, and when the file is released. Each of those decisions has a failure that looks unrelated to opening — a slow mean, a memory spike, an integer where a temperature should be, a file that cannot be overwritten.

Measured on NOAA's NCEP/NCAR Reanalysis 1 monthly air temperature (29.6 MB, 938 × 73 × 144) and one day of NOAA OISST v2.1:

  • Opening read almost nothing. Traced memory after open_dataset was 0.03 MB; after selecting 12 months, 0.54 MB; after reading every value, 39.47 MB.
  • Asking for dask with the file's own chunks made it over three times slower. chunks={} split the file into its 938 one-month chunks, and opening plus a global mean took 529.1 ms against 161.8 ms without dask.
  • The same cell read three ways gave 30.83 °C, 3083 and 3083 depending on the decoding switches.
  • Over HTTP, opening the file by byte ranges took 185.8 seconds; downloading it whole took 6.06.

Quick answer 

import xarray as xr

with xr.open_dataset("air.mon.mean.nc") as ds:          # lazy: header and coordinates only
    july = ds["air"].sel(time="2024-07").load()          # read just what you need
print(dict(july.sizes), float(july.mean()))
{'time': 1, 'lat': 73, 'lon': 144} 7.860571384429932

Use open_dataset without chunks for files that fit in memory, pass chunks={"time": ...} sized to your work when they do not, and select before you load. The with block closes the file as soon as the data you need are in memory.

Bar chart of the time to open the NCEP monthly file and compute a global mean with no dask, chunks equal to the file's chunks, automatic chunks and 120-month chunks.
Dask helps when chunks are large enough to be worth scheduling; one-month chunks were not.

Step-by-step solution

1. Check which engines are installed

xarray reads NetCDF through a backend engine. This environment listed netcdf4, h5netcdf, scipy, cfgrib, rasterio, store and zarr, and tries netcdf4, then h5netcdf, then scipy for a NetCDF file. netcdf4 reads both the classic and the NetCDF-4 format; scipy reads only the classic format and refused the NetCDF-4 monthly file; h5netcdf needs the h5py package, and without it raised ImportError: No module named 'h5py'. When no engine matches, xarray raises the error covered in fixing "did not find a match in any of xarray's installed backends".

2. Open lazily and look before loading

open_dataset reads the header, the coordinates and the attributes, and wraps each data variable in a lazy array. Printing the dataset shows dimensions, coordinates, variables and attributes without touching the data. Opening took 5.3 ms; loading the whole file took 42.1 ms.

3. Pick the variable

ds["air"] gives the data variable as a DataArray. xr.open_dataarray is a shortcut for single-variable files only: on the four-variable OISST file it raised ValueError: Given file dataset contains more than one data variable.

4. Decide whether you need dask

Without chunks, xarray reads with NumPy when you compute, which is fastest for anything that fits in memory. Passing chunks creates a dask array:

option                          open + global mean   dask chunks
no chunks (NumPy)                   161.8 ms             —
chunks={}  (file's own chunks)      529.1 ms            938
chunks="auto"                       166.1 ms              1
chunks={"time": 120}                136.9 ms              8

These are the fastest of five runs of Example 2; absolute times vary with the machine and its load, the ordering did not.

chunks={} follows the chunking stored in the file, which here is one month per chunk: 938 tiny tasks whose scheduling cost dwarfed the work. Choose chunk sizes from the analysis — a few hundred megabytes each is a common target — not from the file. See lazy loading explained.

5. Control decoding when you need the stored values

One OISST ocean cell, one land cell and the time, read three ways:

option                 dtype     ocean cell   land cell   time
default                float32   30.83        NaN         2024-01-15T12:00
mask_and_scale=False   int16     3083         -999        2024-01-15T12:00
decode_cf=False        int16     3083         -999        16815.0

The default is almost always right. Switch decoding off to inspect packing and fill values, and only then — see NetCDF values that look wrong.

6. Select, then load

Everything stays lazy until you call .load(), .values, .compute(), plot or write. Selecting 12 months of the monthly file and loading them used 0.54 MB; reading every value used 39.47 MB. On a file larger than memory, that ordering is the difference between working and failing.

7. Close the file

A dataset keeps its file open until it is closed or garbage-collected. Writing a subset back to the same path while the original was still open failed with PermissionError: [Errno 13] Permission denied. Use with xr.open_dataset(...), or call .load() and .close() before writing to the same path — or write to a new path.

8. Download remote files before opening them

The netCDF library can open an HTTPS URL with #mode=bytes, fetching pieces of the file on demand. For the 29.6 MB monthly file, opening took 185.8 seconds, one map 0.89 seconds and a 120-month series for one cell 22.11 seconds, because HDF5 metadata is scattered through the file and each piece is a separate request. Downloading the whole file took 6.06 seconds. Formats designed for the cloud, such as Zarr, avoid this.

Table of one OISST ocean cell, one land cell and the time value under default decoding, mask_and_scale=False and decode_cf=False.
Decoding is on by default; the switches exist for inspecting a file, not for analysis.

Code examples

Example 1 — describe a file before using it 

import xarray as xr


def describe(path):
    with xr.open_dataset(path) as ds:
        print(path)
        print(f"  dimensions {dict(ds.sizes)}")
        if "time" in ds.coords:
            print(f"  time {str(ds.time.values[0])[:10]} to {str(ds.time.values[-1])[:10]}")
        for name, var in ds.data_vars.items():
            enc = var.encoding
            print(f"  {name:4} {var.dims} {var.dtype}, {var.nbytes / 1e6:.1f} MB in memory; "
                  f"stored as {enc.get('dtype')}, scale {enc.get('scale_factor', '-')}, chunks {enc.get('chunksizes')}")


describe("air.mon.mean.nc")
describe("oisst-avhrr-v02r01.20240115.nc")
air.mon.mean.nc
  dimensions {'time': 938, 'lat': 73, 'lon': 144}
  time 1948-01-01 to 2026-02-01
  air  ('time', 'lat', 'lon') float32, 39.4 MB in memory; stored as float32, scale 1.0, chunks (1, 73, 144)
oisst-avhrr-v02r01.20240115.nc
  dimensions {'time': 1, 'zlev': 1, 'lat': 720, 'lon': 1440}
  time 2024-01-15 to 2024-01-15
  sst  ('time', 'zlev', 'lat', 'lon') float32, 4.1 MB in memory; stored as int16, scale 0.009999999776482582, chunks (1, 1, 720, 1440)
  anom ('time', 'zlev', 'lat', 'lon') float32, 4.1 MB in memory; stored as int16, scale 0.009999999776482582, chunks (1, 1, 720, 1440)
  err  ('time', 'zlev', 'lat', 'lon') float32, 4.1 MB in memory; stored as int16, scale 0.009999999776482582, chunks (1, 1, 720, 1440)
  ice  ('time', 'zlev', 'lat', 'lon') float32, 4.1 MB in memory; stored as int16, scale 0.009999999776482582, chunks (1, 1, 720, 1440)

The scale factor prints as 0.009999999776482582 because it is stored as a 32-bit float.

Example 2 — measure the chunking options on your own file 

import time


def open_and_mean(path, variable, **kwargs):
    start = time.perf_counter()
    with xr.open_dataset(path, **kwargs) as ds:
        data = ds[variable]
        tasks = data.data.npartitions if hasattr(data.data, "npartitions") else None
        float(data.mean())
    return (time.perf_counter() - start) * 1000, tasks


for label, kwargs in [("no chunks", {}), ("chunks={}", {"chunks": {}}),
                      ("chunks='auto'", {"chunks": "auto"}), ("chunks={'time': 120}", {"chunks": {"time": 120}})]:
    runs = [open_and_mean("air.mon.mean.nc", "air", **kwargs) for _ in range(5)]
    print(f"{label:22} {min(ms for ms, _ in runs):7.1f} ms   dask chunks {runs[0][1]}")
no chunks                161.8 ms   dask chunks None
chunks={}                529.1 ms   dask chunks 938
chunks='auto'            166.1 ms   dask chunks 1
chunks={'time': 120}     136.9 ms   dask chunks 8

Run it on the files and the operation you actually have: the right answer depends on the file's stored chunks, its size and the computation.

Example 3 — refuse a file that is not NetCDF before xarray guesses 

MAGIC = {b"CDF\x01": "NetCDF classic", b"CDF\x02": "NetCDF 64-bit offset",
         b"CDF\x05": "NetCDF 64-bit data", b"\x89HDF\r\n\x1a\n": "NetCDF-4 / HDF5"}


def netcdf_kind(path):
    with open(path, "rb") as f:
        head = f.read(8)
    for magic, kind in MAGIC.items():
        if head.startswith(magic):
            return kind
    raise ValueError(f"{path} is not a NetCDF file; it starts with {head!r}")


with open("download.nc", "w") as f:
    f.write("<html><head><title>429 Too Many Requests</title></head></html>")
xr.Dataset({"v": ("x", [1.0, 2.0])}).to_netcdf("classic.nc", format="NETCDF3_CLASSIC")

for path in ["air.mon.mean.nc", "classic.nc", "download.nc"]:
    try:
        print(path, "->", netcdf_kind(path))
    except ValueError as error:
        print(error)
air.mon.mean.nc -> NetCDF-4 / HDF5
classic.nc -> NetCDF classic
download.nc is not a NetCDF file; it starts with b'<html><h'

Handed the same HTML file, xr.open_dataset raised ValueError: did not find a match in any of xarray's currently installed IO backends.

A failed download saved with a .nc name is a common cause of confusing backend errors in batch jobs; checking the first bytes turns it into a clear message.

Explanation

Why opening is lazy

A NetCDF header lists every variable's name, type, shape and location in the file, so xarray can build the whole dataset structure without reading data. The arrays are wrapped in lazy indexing objects that translate a selection into a read of just the chunks it needs. That is what lets a 29.6 MB or a 29.6 GB file open in milliseconds.

Why chunks={} was slow

Dask turns each chunk into a task and schedules them. For large chunks the scheduling cost is negligible; for 938 chunks of 73 × 144 floats — about 30 kB each — it dominated. The file's chunks were chosen for storage and for reading one map at a time, not for computing a global mean.

Why decoding is on by default

Packed integers, fill values and numeric time offsets are storage details. With decoding on, arithmetic works on physical values and missing data are NaN; with it off, a mean mixes −999 land values into sea temperatures. The switches exist so you can see what is stored when decoding produces something unexpected.

Why files stay open

xarray keeps a handle so that later selections can read from the file. HDF5 locks files that are open, which is why writing to the same path fails. A context manager ties the handle's lifetime to a block of code.

Bar chart of traced memory after opening the NCEP monthly file, after selecting 12 months and after reading all values.
Memory follows what you read, not what you open.

Edge cases or notes

  • Multi-file datasets use open_mfdataset; see opening hundreds of NetCDF files.
  • Groups in NetCDF-4 files need group="name"; the root group is opened by default.
  • GRIB files open with engine="cfgrib", with their own pitfalls; see reading GRIB data.
  • Non-standard calendars decode to cftime objects rather than NumPy datetimes.
  • Re-saving an opened file carries its original encoding with it, including packing and any precision-reducing settings.
  • decode_times=False is useful when times fail to decode, to see the raw units and calendar.
  • File caching keeps up to a set number of files open; very large multi-file jobs can hit the operating system's open-file limit.

FAQ

How do I open a NetCDF file in Python?

With xarray: xr.open_dataset("file.nc"), then select a variable with ds["name"]. It needs a backend such as netCDF4 installed; opening reads only the header and coordinates.

Does xarray open_dataset load the whole file into memory?

No. Traced memory after opening the 29.6 MB monthly file was 0.03 MB; it rose to 39.47 MB only when every value was read.

Should I use chunks when opening a NetCDF file?

Only when the data or the computation does not fit in memory, and then with chunks sized to the work. Using the file's own one-month chunks made a global mean over three times slower.

What is the difference between open_dataset and open_dataarray?

open_dataset returns every variable; open_dataarray returns a single DataArray and raises a ValueError on files with more than one data variable.

Why can I not overwrite a NetCDF file I opened with xarray?

The file is still open and locked. Writing back to the same path failed with a PermissionError; close the dataset first, or write to a new file.

How do I open a NetCDF file from a URL?

Download it first when you can. Reading the monthly file by HTTP byte ranges took 185.8 seconds to open, against 6.06 seconds to download it whole.