⚙️ How-To Gridded & climate data Batch processing Performance & scale Gridded & Climate Data · step 15 of 20 Updated

How to Open Hundreds of NetCDF Files as One Dataset

Problem statement

Daily and monthly gridded products usually arrive as one file per time step: 366 files for a year of daily sea surface temperature, thousands for a decade. xr.open_mfdataset stitches them into one lazy dataset, and on the first try it usually works. What it does behind that call — opening every file, checking every coordinate, building a task per chunk — decides how long each later calculation takes, how much memory it uses, and whether it works at all when run in parallel.

Measured on the 366 daily NOAA OISST v2.1 files for 2024 (560 MB), computing the daily mean over a North Atlantic box:

  • open_mfdataset took 2.1 s to open the year, created 1,464 dask tasks for sea surface temperature alone, and the box-mean series took a further 2.6 s.
  • Opening each file in a loop and loading only the box took 5.0 s in total but peaked at 271 MB of memory against 843 MB.
  • parallel=True failed in all six attempts: five killed Python with double-free errors or segmentation faults, the sixth raised an HDF error.
  • Converted once to Zarr — 3.7 to 7.6 s, 355 MB in 22 files — the year reopened in 0.36 s, the box series took 0.67 s, and a single cell's series 0.36–0.51 s instead of 2.44–2.77 s.

Quick answer 

import glob

import xarray as xr

files = sorted(glob.glob("oisst-2024/*.nc"))
ds = xr.open_mfdataset(
    files, combine="nested", concat_dim="time",
    data_vars="minimal", coords="minimal", compat="override", join="override",
    drop_variables=["anom", "err", "ice"],
)
box = ds["sst"].sel(lat=slice(40, 60), lon=slice(320, 350)).mean(["lat", "lon"]).compute()
print(dict(ds.sizes), box.sizes["time"])

Sort the file list, tell xarray how the files relate when you know it, open only the variables you need, and convert to Zarr if you will read the collection repeatedly.

Bar chart of time to open 366 daily OISST files and compute a box-mean series with open_mfdataset defaults, fast combine options, one variable, a preprocessing subset, an eager loop and a Zarr copy.
Most open_mfdataset options changed little on identical daily files; converting to Zarr changed everything.

Step-by-step solution

1. List and sort the files yourself

glob.glob returns files in file-system order, which is not guaranteed to be date order. Sort the list, and check its length against the number of time steps you expect — 366 for a leap year. A missing or extra file is far easier to spot here than after combining.

2. Open one file first

Look at one file's dimensions, coordinates and variables before combining them all. Each OISST day holds sst, anom, err and ice on (time, zlev, lat, lon) with lengths 1, 1, 720 and 1440; a single open_dataset took 10 ms.

3. Choose how files are combined

combine="by_coords", the default, reads every file's coordinates and orders files by them; it copes with an unsorted list. combine="nested" with concat_dim="time" concatenates in list order without looking. With a shuffled list, nested combining produced a time axis starting on 29 January that was not monotonic, and a later sel(time=slice(...)) failed; by-coordinate combining sorted it. Use nested only on a list you have sorted and checked.

4. Skip checks only when files are guaranteed identical

data_vars="minimal", coords="minimal", compat="override" and join="override" tell xarray not to compare coordinates and non-concatenated variables across files, and to trust the first file. On OISST's identical daily grids they did not make opening faster — 2.4 s against 2.1 s — because the checks are cheap for 366 small files; they matter for files with large coordinate arrays or many variables. They also remove the protection against one mismatched file; see fixing open_mfdataset that will not combine.

5. Open only what you need

drop_variables skips variables at open time. Dropping three of four variables opened the year in 1.9–2.1 s. A preprocess function runs on each file before combining: selecting the box and the one variable there reduced peak memory from 843 MB to 772 MB, and the series took 2.6 s.

6. Count the tasks before computing

Every file is at least one dask chunk per variable. Opening all four variables gave 1,464 tasks for sst, four per file, before any computation was added; dropping the other variables at open, as in Example 1, gave 1,098, three per file. Hundreds of tiny tasks cost more to schedule than to run; rechunking along time, or reading into memory when the result is small, is often faster.

7. Avoid parallel=True with the netCDF4 library in threads

parallel=True opens files concurrently with dask. With the default threaded scheduler and netCDF4 it failed every time, and five of six attempts killed the process: double free or corruption, NetCDF: HDF error, NetCDF: Can't open HDF5 attribute and segmentation faults. The HDF5 library underneath is not safe for concurrent use in one process. Open sequentially, or use a process-based dask cluster.

8. Convert to Zarr when you will come back

A one-off conversion of sst to Zarr with one chunk per 31 days took 7.6 s in one run and 3.7 s in another, producing 355 MB in 22 files both times. Reopening it took 0.36 s, with 25 tasks instead of 1,464; the box series took 0.67 s, and a single cell's 366-day series 0.36–0.51 s against 2.44–2.77 s from the NetCDF collection. The conversion pays for itself after two or three reads.

Bar chart comparing open time, box-mean compute time and single-cell series time for the 366 NetCDF files and the converted Zarr store.
366 files with four variables each become 22 chunk files of one variable: fewer, larger reads.

Code examples

Example 1 — open a collection and report what it built 

import time

import numpy as np


def open_collection(pattern, variables, expected=None, **options):
    files = sorted(glob.glob(pattern))
    if expected is not None and len(files) != expected:
        raise ValueError(f"{len(files)} files match {pattern!r}, expected {expected}")
    with xr.open_dataset(files[0]) as first:
        drop = [v for v in first.data_vars if v not in variables]
    start = time.perf_counter()
    ds = xr.open_mfdataset(files, drop_variables=drop, **options)
    seconds = time.perf_counter() - start
    t = ds.indexes["time"]
    tasks = {v: len(ds[v].data.__dask_graph__()) for v in ds.data_vars}
    print(f"{len(files)} files in {seconds:.2f} s; time {t[0]:%Y-%m-%d} to {t[-1]:%Y-%m-%d}, "
          f"sorted {t.is_monotonic_increasing}, duplicates {int(t.duplicated().sum())}; tasks {tasks}")
    return ds


ds = open_collection("oisst-2024/*.nc", ["sst"], expected=366)
366 files in 1.55 s; time 2024-01-01 to 2024-12-31, sorted True, duplicates 0; tasks {'sst': 1098}

Example 2 — subset while opening 

BOX = dict(lat=slice(40, 60), lon=slice(320, 350))


def north_atlantic(day):
    return day[["sst"]].sel(**BOX)


start = time.perf_counter()
subset = xr.open_mfdataset(sorted(glob.glob("oisst-2024/*.nc")), preprocess=north_atlantic,
                           combine="nested", concat_dim="time", data_vars="minimal", coords="minimal",
                           compat="override", join="override")
series = subset["sst"].mean(["lat", "lon"]).compute()
print(f"{dict(subset.sizes)} in {time.perf_counter() - start:.2f} s; first day {float(series.isel(time=0).squeeze()):.3f} °C")
{'time': 366, 'zlev': 1, 'lat': 80, 'lon': 120} in 4.35 s; first day 11.951 °C

The preprocess function sees one file at a time, so anything it drops is never read or combined.

Example 3 — convert once to Zarr and read from there 

import os
import shutil


def fastest(fn, repeats=3):
    best = float("inf")
    for _ in range(repeats):
        start = time.perf_counter()
        fn()
        best = min(best, time.perf_counter() - start)
    return best


store = "oisst2024_sst.zarr"
shutil.rmtree(store, ignore_errors=True)
start = time.perf_counter()
converted = ds[["sst"]].chunk({"time": 31})
for name in converted.variables:
    converted[name].encoding = {}
converted.to_zarr(store, mode="w", consolidated=True)
files = [os.path.join(d, f) for d, _, names in os.walk(store) for f in names]
print(f"write {time.perf_counter() - start:.1f} s, {sum(map(os.path.getsize, files)) / 1e6:.1f} MB in {len(files)} files")

z = xr.open_zarr(store)
cell = dict(lat=50.125, lon=330.125)
print(f"point series: NetCDF {fastest(lambda: ds['sst'].sel(**cell).compute()):.2f} s, "
      f"Zarr {fastest(lambda: z['sst'].sel(**cell).compute()):.2f} s")
write 3.7 s, 355.1 MB in 22 files
point series: NetCDF 2.44 s, Zarr 0.36 s

Clearing the NetCDF encoding stops the int16 packing and chunk settings of the source files being carried into the Zarr store.

Explanation

What open_mfdataset does before you compute anything

It opens every file to read its metadata, decides the order and alignment of the files from their coordinates (or from the list order for nested combining), checks that variables shared across files agree, and wraps each file's arrays in dask chunks. None of the data values are read. The cost is proportional to the number of files and variables, not their size.

Why the fast options did not help here

compat="override" and join="override" skip comparisons of coordinates and variables across files. OISST daily files have small, identical 1-D coordinates, so the comparisons were a tiny part of the 2.1 s; the rest was opening 366 HDF5 files. Collections with 2-D coordinate arrays, many auxiliary variables or slightly different grids spend far more time there.

Why parallel opening crashed

parallel=True wraps each open_dataset call in a dask delayed task and runs them concurrently. With threads, several calls enter the HDF5 C library at once, and the builds used here do not serialise those calls safely. The crashes were intermittent in form — double frees, HDF errors, segfaults — which is typical of memory corruption. Process-based parallelism gives each worker its own copy of the library.

Why Zarr reads faster

The NetCDF collection keeps one small chunk per variable per day, each inside a separate HDF5 file that has to be opened. The Zarr store holds only sst, in 12 chunks of 31 days each, with metadata consolidated into one object. A series for one cell touches 12 compressed chunks rather than 366 files.

Table of six attempts to open the OISST year with parallel=True and the error each produced.
Six attempts, six failures, several different error messages: a thread-safety problem, not a data problem.

Edge cases or notes

  • Too many open files: with a limit of 64 file handles the default open failed; xr.set_options(file_cache_maxsize=32) made it succeed.
  • Files with a missing variable are filled with NaN for that file under by-coordinate combining.
  • Duplicate or preliminary files in a folder add duplicate time steps under nested combining.
  • Remote collections multiply the per-file cost by network latency; index them with Kerchunk or convert them.
  • Different grids in one collection need regridding before combining; see regridding explained.
  • GRIB collections use engine="cfgrib" with the same combine options; see reading GRIB data.
  • Appending new days to a Zarr store uses to_zarr(..., append_dim="time").

FAQ

How do I open many NetCDF files at once in xarray?

Use xr.open_mfdataset with a sorted list of files. The 366 daily OISST files for 2024 opened in 2.1 s as one lazy dataset.

Why is open_mfdataset slow?

It opens every file and builds dask tasks for every chunk: 1,464 tasks for one variable across 366 files. Drop unneeded variables, subset with preprocess, or convert the collection to Zarr.

Should I use parallel=True in open_mfdataset?

Not with the netCDF4 library and threads: it crashed in all six attempts here. Use sequential opening or a process-based dask cluster.

What do combine="nested" and concat_dim do?

They concatenate files in list order along the named dimension without inspecting coordinates. It is safe only on a sorted, checked list; a shuffled list produced an unsorted time axis.

When should I convert NetCDF files to Zarr?

When you will read the collection more than a couple of times. Conversion took 3.7–7.6 s; afterwards a single cell's series took at most 0.51 s instead of at least 2.44 s.

How much memory does open_mfdataset use?

Opening and computing a box series over the OISST year peaked at 843 MB; loading each file's box in a loop peaked at 271 MB.