🔧 Fix Gridded & climate data Performance & scale Batch processing Gridded & Climate Data · step 19 of 20 Updated

Fixing open_mfdataset That Is Slow, Hangs or Will Not Combine

Problem statement

xr.open_mfdataset("folder/*.nc") fails in ways that have little to do with the error message. A single cropped file produces a complaint about monotonic indexes. A download error page produces a missing-backend error. Too many files produces the same missing-backend error. A duplicate file produces no error at all, just an extra day. And parallel=True, the obvious speed-up, crashes.

Reproduced with the 31 daily NOAA OISST v2.1 files for January 2024, each time adding one problem:

  • A cropped regional file made by-coordinate combining raise ValueError: Resulting object does not have monotonic global indexes along dimension lon; a latitude shifted by 0.0001° in one file raised the same error for lat.
  • A file with one variable missing combined without error; that variable was filled with NaN for the whole day.
  • A duplicate "preliminary" copy of one day gave 32 time steps with a duplicate under nested combining; by-coordinate combining returned 31 without warning.
  • An HTML error page saved as .nc raised did not find a match in any of xarray's currently installed IO backends — the same message produced, for the full year, by a limit of 64 open files.
  • parallel=True on the 366-file year failed in all six attempts: five killed the interpreter with double frees or segmentation faults, the sixth raised an HDF error.

Quick answer 

symptom                                                    cause                                   fix
"does not have monotonic global indexes along dimension"   one file on a different or shifted grid  find it (Example 1); regrid or drop
"did not find a match in any ... installed IO backends"    a non-NetCDF file, or too many open files check file headers; lower file_cache_maxsize
OSError: NetCDF: HDF error                                 truncated or corrupt file                re-download it
extra or duplicated time steps, no error                   duplicate files in the folder            check time duplicates after opening
a variable all NaN for some days, no error                 files with different variables           check variable sets per file
crash, "double free", segfault with parallel=True          HDF5 not thread-safe in this build       drop parallel=True or use processes
"Could not find any dimension coordinates to use"          files without the time coordinate        use nested combining with concat_dim

Then scan the folder before opening it (Example 1), and open with join="exact" so mismatches fail immediately instead of being padded.

Triage of open_mfdataset failures: non-monotonic index errors, missing backend errors, HDF errors, silent duplicates, silent missing variables and parallel crashes, each with its cause and fix.
Half of these failures raise misleading errors; the other half raise none.

Step-by-step solution

1. Check the glob matched what you expect

open_mfdataset("nothing_here/*.nc") raised OSError: no files to open. A glob that matched too many files is quieter: a stray regional extract, a preliminary copy or a partial download all match *.nc. Count the files and compare with the number of days.

2. Check that every file is NetCDF

A download that returned an HTML error page, saved under a .nc name, made the whole open fail with ValueError: did not find a match in any of xarray's currently installed IO backends. A truncated download raised OSError: [Errno -101] NetCDF: HDF error. Checking each file's first bytes — CDF or \x89HDF — names the bad file directly (Example 1).

3. Find the file on a different grid

By-coordinate combining sorts files by their coordinates and requires one consistent grid. One file cropped to the North Atlantic, or with latitudes shifted by 0.0001°, produced a ValueError about monotonic global indexes along lon or lat — the error describes the combined result, not the file responsible. Listing each file's grid size and first coordinate values identifies it. With join="exact", the cropped file raised AlignmentError: cannot align objects with join='exact' instead; with join="override" it still failed because the sizes differed.

4. Check variables per file

A file without the ice variable combined silently: by-coordinate combining filled ice with NaN for that day, and nested combining also succeeded without a message. Compare the set of data variables in each file before combining, or open with data_vars limited to the variables you need.

5. Check time after opening

Nested combining trusts list order and does not look for duplicates: a folder with a preliminary copy of 15 January gave 32 time steps and one duplicate. A shuffled file list gave a time axis that was not monotonic, and slicing it raised KeyError: 'Value based partial slicing on non-monotonic DatetimeIndexes with non-existing keys is not allowed.' After any multi-file open, assert that time is sorted and unique.

6. Handle files without a time coordinate

Files that store a single field with no time variable cannot be ordered by coordinates: one such file raised ValueError: Every dimension requires a corresponding 1D coordinate and index for inferring concatenation order but the coordinate 'time' has no corresponding index, and a folder of them Could not find any dimension coordinates to use to order the Dataset objects for concatenation. Use combine="nested" with a sorted list and add time afterwards, or add it in preprocess from the file name.

7. Raise the open-file limit or shrink the cache

xarray keeps up to 128 files open by default. With the process limited to 64 open files, opening the 366-file year failed — surprisingly, with the "did not find a match in any of xarray's currently installed IO backends" error, because the backends could not open the files to identify them. xr.set_options(file_cache_maxsize=32) fixed it, and the mean computed as 14.13 °C.

8. Do not use parallel=True with the netCDF4 library in threads

On the full 366-file year, parallel=True failed in six of six attempts, five of them killing the process: double free or corruption, malloc(): unaligned tcache chunk detected, NetCDF: HDF error, NetCDF: Can't open HDF5 attribute and two segmentation faults. Sequential opening took 2.1 s. If opening is the bottleneck, convert the collection to Zarr once — see opening hundreds of NetCDF files.

Two panels showing the failures that raised no error: a duplicate daily file producing an extra time step, and a file missing a variable producing a day of NaN.
The worst multi-file problems are the ones that combine successfully.

Code examples

Example 1 — scan a folder before combining 

import glob
import os

import netCDF4
import pandas as pd


def scan(pattern):
    rows = []
    for path in sorted(glob.glob(pattern)):
        row = {"file": os.path.basename(path)}
        with open(path, "rb") as f:
            head = f.read(8)
        if not (head.startswith(b"CDF") or head.startswith(b"\x89HDF")):
            row["problem"] = f"not NetCDF: starts {head[:6]!r}"
        else:
            try:
                with netCDF4.Dataset(path) as nc:
                    row["variables"] = ",".join(sorted(v for v, var in nc.variables.items() if var.ndim >= 3))
                    row["grid"] = f"{len(nc.dimensions['lat'])}x{len(nc.dimensions['lon'])} from {float(nc['lat'][0]):.4f}"
                    row["time"] = float(nc["time"][0]) if "time" in nc.variables else None
            except OSError as error:
                row["problem"] = str(error).split(":")[0] + ": " + str(error).split(":")[1].strip()
        rows.append(row)
    table = pd.DataFrame(rows)
    print(f"{len(table)} files")
    for column in ("problem", "variables", "grid"):
        if column in table:
            counts = table[column].value_counts(dropna=True)
            if column == "problem" or len(counts) > 1:
                print(f"  {column}:", counts.to_dict())
    if "time" in table:
        timed = table.dropna(subset=["time"])
        if timed.time.duplicated(keep=False).any():
            print("  duplicate times:", timed.loc[timed.time.duplicated(keep=False), "file"].tolist())
    return table


table = scan("broken/*.nc")
35 files
  problem: {"not NetCDF: starts b'<!DOCT'": 1, '[Errno -101] NetCDF: HDF error': 1}
  grid: {'720x1440 from -89.8750': 32, '280x320 from 0.1250': 1}
  duplicate times: ['oisst-avhrr-v02r01.20240115.nc', 'oisst-avhrr-v02r01.20240115_preliminary.nc']

The folder held January plus four planted problems: a preliminary copy of 15 January, an HTML page and a truncated download named as 1 and 2 February, and a cropped 3 February. The scan names each one.

Example 2 — open strictly and check the result 

import xarray as xr


def open_strict(files, dim="time"):
    ds = xr.open_mfdataset(files, combine="by_coords", join="exact", data_vars="minimal", coords="minimal")
    index = ds.indexes[dim]
    if not index.is_monotonic_increasing:
        raise ValueError(f"{dim} is not sorted")
    if index.has_duplicates:
        raise ValueError(f"{int(index.duplicated().sum())} duplicate {dim} values")
    empty = [v for v in ds.data_vars if bool(ds[v].isnull().all(dim=[d for d in ds[v].dims if d != dim]).any())]
    if empty:
        raise ValueError(f"variables entirely missing on some {dim} steps: {empty}")
    return ds


for folder in ("clean", "cropped", "missing_variable"):
    try:
        ds = open_strict(sorted(glob.glob(f"{folder}/*.nc")))
        print(f"{folder}: ok {dict(ds.sizes)}")
    except Exception as error:
        print(f"{folder}: {type(error).__name__}: {str(error).splitlines()[0][:110]}")
clean: ok {'time': 31, 'zlev': 1, 'lat': 720, 'lon': 1440}
cropped: AlignmentError: cannot align objects with join='exact' where index/labels/sizes are not equal along these coordinates (dimensi
missing_variable: AlignmentError: cannot align objects with join='exact' where index/labels/sizes are not equal along these coordinates (dimensi

With join="exact" the folder with a missing variable failed as well, where the default join had combined it silently, so the NaN check after opening was never reached.

Example 3 — too many open files 

import subprocess
import sys

code = ("import xarray as xr; {opt} ds = xr.open_mfdataset('oisst-2024/*.nc'); "
        "print('mean', round(float(ds.sst.mean()), 2))")
for option in ("", "xr.set_options(file_cache_maxsize=32);"):
    result = subprocess.run(["bash", "-c", f'ulimit -n 64; {sys.executable} -W ignore -c "{code.format(opt=option)}"'],
                            capture_output=True, text=True)
    message = result.stdout.strip() or [line for line in result.stderr.splitlines() if "Error" in line][-1]
    print(f"{option or 'default file cache':42} exit {result.returncode}: {message[:100]}")
default file cache                         exit 1: ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'h5n
xr.set_options(file_cache_maxsize=32);     exit 0: mean 14.13

ulimit -n 64 imitates a restrictive environment such as a batch job or a container; on a typical workstation the limit is far higher, so the failure appears only on larger collections.

Explanation

Why the errors point at the wrong thing

open_mfdataset does not report which file caused a problem. A cropped file makes the concatenated longitude index non-monotonic, so the error describes the index. A file no backend can read makes backend detection fail, so the error describes the backends. A process that cannot open more files makes the same detection fail. Scanning files individually turns each of those into a named file and a specific problem.

Why some problems raise nothing

xarray is designed to combine imperfect data: an outer join pads missing coordinates, a missing variable becomes NaN, nested concatenation keeps every step it is given. Those defaults keep workflows running, and they mean a duplicate or incomplete file changes the data without any message. Strict options — join="exact" and explicit checks after opening — trade that convenience for early failure.

Why the default join is changing

Opening several of these folders printed FutureWarning: In a future version of xarray the default value for join will change from join='outer' to join='exact'. When that change lands, mismatched grids will fail instead of being padded — the behaviour Example 2 opts into now.

Why parallel opening is unsafe here

Each file open enters the HDF5 C library. The builds used here are not compiled with thread safety, so concurrent opens from dask's thread pool corrupt shared state. The symptoms vary from run to run because they depend on timing, which is why a crash in one attempt and an HDF error in the next point at the same cause.

Flow for diagnosing a folder of NetCDF files: count files, check headers, compare grids and variables, open strictly, then check time for order and duplicates.
Five checks, in order of how cheap they are; each one names the file responsible.

Edge cases or notes

  • Hangs rather than errors usually mean a network file system or a remote URL; test with a local copy first.
  • Hundreds of thousands of files make even the metadata scan slow; build a Kerchunk index or convert.
  • Different chunking between files is allowed but produces uneven dask chunks.
  • Mixed calendars across model files fail when times are combined; see fixing cftime errors.
  • Engine-specific failures can be isolated by passing engine="netcdf4" explicitly.
  • Compressed .nc.gz files are not NetCDF until decompressed.
  • Upcoming default changes to join, compat and data_vars will turn some silent cases into errors; set them explicitly to keep behaviour stable.

FAQ

Why does open_mfdataset say the index is not monotonic?

One file has a different or shifted grid. A single cropped OISST file raised the error for longitude; a latitude shifted by 0.0001° raised it for latitude. Scan the files' grids to find it.

Why does open_mfdataset say no backend matches when netCDF4 is installed?

One file is not NetCDF — often an HTML error page from a failed download — or the process cannot open more files. Both produced the same message here.

Why does my combined dataset have duplicate dates?

A duplicate or preliminary file is in the folder. Nested combining kept both copies of the day; check index.has_duplicates after opening.

How do I fix "too many open files" in open_mfdataset?

Lower xarray's file cache with xr.set_options(file_cache_maxsize=...) or raise the operating system limit. With a 64-file limit, a cache of 32 let the 366-file year open.

Why does parallel=True crash open_mfdataset?

The HDF5 library underneath netCDF4 is not safe for concurrent use in threads in these builds. Six attempts crashed; open sequentially or use separate processes.

How do I make open_mfdataset fail loudly on mismatched files?

Pass join="exact" and check the time index for order and duplicates after opening. The cropped file then raised an alignment error instead of being padded.