Fixing open_mfdataset That Is Slow, Hangs or Will Not Combine

Problem statement

xr.open_mfdataset("folder/*.nc") fails in ways that have little to do with the error message. A single cropped file produces a complaint about monotonic indexes. A download error page produces a missing-backend error. Too many files produces the same missing-backend error. A duplicate file produces no error at all, just an extra day. And parallel=True, the obvious speed-up, crashes.

Reproduced with the 31 daily NOAA OISST v2.1 files for January 2024, each time adding one problem:

• A cropped regional file made by-coordinate combining raise ValueError: Resulting object does not have monotonic global indexes along dimension lon; a latitude shifted by 0.0001° in one file raised the same error for lat.
• A file with one variable missing combined without error; that variable was filled with NaN for the whole day.
• A duplicate "preliminary" copy of one day gave 32 time steps with a duplicate under nested combining; by-coordinate combining returned 31 without warning.
• An HTML error page saved as .nc raised did not find a match in any of xarray's currently installed IO backends — the same message produced, for the full year, by a limit of 64 open files.
• parallel=True on the 366-file year failed in all six attempts, and every one killed the interpreter with a segmentation fault. The last message before the crash changed from run to run: NetCDF: HDF error, double free or corruption, or nothing.

Quick answer

symptom                                                    cause                                   fix
"does not have monotonic global indexes along dimension"   one file on a different or shifted grid  find it (Example 1); regrid or drop
"did not find a match in any ... installed IO backends"    a non-NetCDF file, or too many open files check file headers; lower file_cache_maxsize
OSError: NetCDF: HDF error                                 truncated or corrupt file                re-download it
extra or duplicated time steps, no error                   duplicate files in the folder            check time duplicates after opening
a variable all NaN for some days, no error                 files with different variables           check variable sets per file
crash, "double free", segfault with parallel=True          HDF5 not thread-safe in this build       drop parallel=True or use processes
"Could not find any dimension coordinates to use"          files without the time coordinate        use nested combining with concat_dim

Then scan the folder before opening it (Example 1), and open with join="exact" so mismatches fail immediately instead of being padded.

Glob, file types, grids, variables and time

1. Check the glob matched what you expect

open_mfdataset("nothing_here/*.nc") raised OSError: no files to open. A glob that matched too many files is quieter: a stray regional extract, a preliminary copy or a partial download all match *.nc. Count the files and compare with the number of days.

2. Check that every file is NetCDF

A download that returned an HTML error page, saved under a .nc name, made the whole open fail with ValueError: did not find a match in any of xarray's currently installed IO backends. A truncated download raised OSError: [Errno -101] NetCDF: HDF error. Checking each file's first bytes — CDF or \x89HDF — names the bad file directly (Example 1).

3. Find the file on a different grid

By-coordinate combining sorts files by their coordinates and requires one consistent grid. One file cropped to the North Atlantic, or with latitudes shifted by 0.0001°, produced a ValueError about monotonic global indexes along lon or lat — the error describes the combined result, not the file responsible. Listing each file's grid size and first coordinate values identifies it. With join="exact", the cropped file raised AlignmentError: cannot align objects with join='exact' instead; with join="override" it still failed because the sizes differed.

4. Check variables per file

A file without the ice variable combined silently: by-coordinate combining filled ice with NaN for that day, and nested combining also succeeded without a message. Compare the set of data variables in each file before combining, or open with data_vars limited to the variables you need.

5. Check time after opening

Nested combining trusts list order and does not look for duplicates: a folder with a preliminary copy of 15 January gave 32 time steps and one duplicate. A shuffled file list gave a time axis that was not monotonic, and slicing it raised KeyError: 'Value based partial slicing on non-monotonic DatetimeIndexes with non-existing keys is not allowed.' After any multi-file open, assert that time is sorted and unique.

6. Handle files without a time coordinate

Files that store a single field with no time variable cannot be ordered by coordinates: one such file raised ValueError: Every dimension requires a corresponding 1D coordinate and index for inferring concatenation order but the coordinate 'time' has no corresponding index, and a folder of them Could not find any dimension coordinates to use to order the Dataset objects for concatenation. Use combine="nested" with a sorted list and add time afterwards, or add it in preprocess from the file name.

7. Raise the open-file limit or shrink the cache

xarray keeps up to 128 files open by default. With the process limited to 64 open files, opening the 366-file year failed — surprisingly, with the "did not find a match in any of xarray's currently installed IO backends" error, because the backends could not open the files to identify them. xr.set_options(file_cache_maxsize=32) fixed it, and the mean computed as 14.13 °C.

8. Do not use parallel=True with the netCDF4 library in threads

On the full 366-file year, parallel=True failed in six of six attempts, and all six killed the process with a segmentation fault; the message before it varied between runs — NetCDF: HDF error, double free or corruption, or none (Example 4). Sequential opening took 1.5 s. If opening is the bottleneck, convert the collection to Zarr once — see opening hundreds of NetCDF files.

Scanning OISST files before combining them

The data used on this page

The files are NOAA's OISST v2.1 daily sea surface temperature for 2024 — 366 files, one per day, each holding sst, anom, err and ice on a 0.25° grid — downloaded from NCEI into oisst-2024/.

import datetime
import os

import requests

BASE = "https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr"
os.makedirs("oisst-2024", exist_ok=True)
day = datetime.date(2024, 1, 1)
while day.year == 2024:
    stamp = day.strftime("%Y%m%d")
    target = f"oisst-2024/oisst-avhrr-v02r01.{stamp}.nc"
    if not os.path.exists(target):
        for name in (f"oisst-avhrr-v02r01.{stamp}.nc", f"oisst-avhrr-v02r01.{stamp}_preliminary.nc"):
            reply = requests.get(f"{BASE}/{day:%Y%m}/{name}", timeout=120)
            if reply.ok:
                open(target, "wb").write(reply.content)
                break
        else:
            raise RuntimeError(f"no OISST file for {stamp}")
    day += datetime.timedelta(days=1)
files = sorted(os.listdir("oisst-2024"))
print(len(files), "daily files,", round(sum(os.path.getsize(f"oisst-2024/{f}") for f in files) / 1e6), "MB")

366 daily files, 560 MB

The failing folders are built from those files. clean/ is January 2024. broken/ adds four planted problems: a copy of 15 January named as its preliminary version (NCEI has since removed the real preliminary files), NCEI's own 404 page saved as 1 February, the first 100 kB of 2 February, and 3 February cropped to a North Atlantic box. cropped/ is January plus that box, and missing_variable/ is January with ice removed from 10 January.

import shutil

import requests
import xarray as xr

JAN = sorted(f"oisst-2024/oisst-avhrr-v02r01.202401{d:02d}.nc" for d in range(1, 32))
for folder in ("clean", "broken", "cropped", "missing_variable"):
    shutil.rmtree(folder, ignore_errors=True)
    os.makedirs(folder)
    for path in JAN:
        os.link(path, f"{folder}/{os.path.basename(path)}")

shutil.copy("oisst-2024/oisst-avhrr-v02r01.20240115.nc", "broken/oisst-avhrr-v02r01.20240115_preliminary.nc")
page = requests.get(f"{BASE}/202402/no-such-file.nc", timeout=60)
open("broken/oisst-avhrr-v02r01.20240201.nc", "wb").write(page.content)
with open("oisst-2024/oisst-avhrr-v02r01.20240202.nc", "rb") as src:
    open("broken/oisst-avhrr-v02r01.20240202.nc", "wb").write(src.read(100_000))
with xr.open_dataset("oisst-2024/oisst-avhrr-v02r01.20240203.nc") as day:
    box = day.sel(lat=slice(0, 70), lon=slice(280, 360)).load()
for folder in ("broken", "cropped"):
    box.to_netcdf(f"{folder}/oisst-avhrr-v02r01.20240203.nc")
os.remove("missing_variable/oisst-avhrr-v02r01.20240110.nc")
with xr.open_dataset("oisst-2024/oisst-avhrr-v02r01.20240110.nc") as day:
    day.drop_vars("ice").load().to_netcdf("missing_variable/oisst-avhrr-v02r01.20240110.nc")
print({folder: len(os.listdir(folder)) for folder in ("clean", "broken", "cropped", "missing_variable")},
      "| 404 page status", page.status_code)

{'clean': 31, 'broken': 35, 'cropped': 32, 'missing_variable': 31} | 404 page status 404

Example 1 — scan a folder before combining

import glob
import os

import netCDF4
import pandas as pd


def scan(pattern):
    rows = []
    for path in sorted(glob.glob(pattern)):
        row = {"file": os.path.basename(path)}
        with open(path, "rb") as f:
            head = f.read(8)
        if not (head.startswith(b"CDF") or head.startswith(b"\x89HDF")):
            row["problem"] = f"not NetCDF: starts {head[:6]!r}"
        else:
            try:
                with netCDF4.Dataset(path) as nc:
                    row["variables"] = ",".join(sorted(v for v, var in nc.variables.items() if var.ndim >= 3))
                    row["grid"] = f"{len(nc.dimensions['lat'])}x{len(nc.dimensions['lon'])} from {float(nc['lat'][0]):.4f}"
                    row["time"] = float(nc["time"][0]) if "time" in nc.variables else None
            except OSError as error:
                row["problem"] = str(error).split(":")[0] + ": " + str(error).split(":")[1].strip()
        rows.append(row)
    table = pd.DataFrame(rows)
    print(f"{len(table)} files")
    for column in ("problem", "variables", "grid"):
        if column in table:
            counts = table[column].value_counts(dropna=True)
            if column == "problem" or len(counts) > 1:
                print(f"  {column}:", counts.to_dict())
    if "time" in table:
        timed = table.dropna(subset=["time"])
        if timed.time.duplicated(keep=False).any():
            print("  duplicate times:", timed.loc[timed.time.duplicated(keep=False), "file"].tolist())
    return table


table = scan("broken/*.nc")

35 files
  problem: {"not NetCDF: starts b'<!DOCT'": 1, '[Errno -101] NetCDF: HDF error': 1}
  grid: {'720x1440 from -89.8750': 32, '280x320 from 0.1250': 1}
  duplicate times: ['oisst-avhrr-v02r01.20240115.nc', 'oisst-avhrr-v02r01.20240115_preliminary.nc']

The folder held January plus four planted problems: a preliminary copy of 15 January, an HTML page and a truncated download named as 1 and 2 February, and a cropped 3 February. The scan names each one.

Example 2 — open strictly and check the result

import xarray as xr


def open_strict(files, dim="time"):
    ds = xr.open_mfdataset(files, combine="by_coords", join="exact", data_vars="minimal", coords="minimal")
    index = ds.indexes[dim]
    if not index.is_monotonic_increasing:
        raise ValueError(f"{dim} is not sorted")
    if index.has_duplicates:
        raise ValueError(f"{int(index.duplicated().sum())} duplicate {dim} values")
    empty = [v for v in ds.data_vars if bool(ds[v].isnull().all(dim=[d for d in ds[v].dims if d != dim]).any())]
    if empty:
        raise ValueError(f"variables entirely missing on some {dim} steps: {empty}")
    return ds


for folder in ("clean", "cropped", "missing_variable"):
    try:
        ds = open_strict(sorted(glob.glob(f"{folder}/*.nc")))
        print(f"{folder}: ok {dict(ds.sizes)}")
    except Exception as error:
        print(f"{folder}: {type(error).__name__}: {str(error).splitlines()[0][:110]}")

clean: ok {'time': 31, 'zlev': 1, 'lat': 720, 'lon': 1440}
cropped: AlignmentError: cannot align objects with join='exact' where index/labels/sizes are not equal along these coordinates (dimensi
missing_variable: AlignmentError: cannot align objects with join='exact' where index/labels/sizes are not equal along these coordinates (dimensi

With join="exact" the folder with a missing variable failed as well, where the default join had combined it silently, so the NaN check after opening was never reached.

Example 3 — too many open files

import subprocess
import sys

code = ("import xarray as xr; {opt} ds = xr.open_mfdataset('oisst-2024/*.nc'); "
        "print('mean', round(float(ds.sst.mean()), 2))")
for option in ("", "xr.set_options(file_cache_maxsize=32);"):
    result = subprocess.run(["bash", "-c", f'ulimit -n 64; {sys.executable} -W ignore -c "{code.format(opt=option)}"'],
                            capture_output=True, text=True)
    message = result.stdout.strip() or [line for line in result.stderr.splitlines() if "Error" in line][-1]
    print(f"{option or 'default file cache':42} exit {result.returncode}: {message[:100]}")

default file cache                         exit 1: ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'h5n
xr.set_options(file_cache_maxsize=32);     exit 0: mean 14.13

ulimit -n 64 imitates a restrictive environment such as a batch job or a container; on a typical workstation the limit is far higher, so the failure appears only on larger collections.

Example 4 — every failure on this page, reproduced

import random
import time


def attempt(label, fn):
    try:
        result = fn()
        print(f"{label:34} ok: {result}")
    except Exception as error:
        message = str(error).replace(os.getcwd() + os.sep, "").splitlines()[0]
        print(f"{label:34} {type(error).__name__}: {message[:95]}")


def jan_plus(extra, name):
    folder = f"try_{name}"
    shutil.rmtree(folder, ignore_errors=True)
    os.makedirs(folder)
    for path in JAN:
        os.link(path, f"{folder}/{os.path.basename(path)}")
    extra(folder)
    return sorted(glob.glob(f"{folder}/*.nc"))


def shifted(folder):
    path = f"{folder}/oisst-avhrr-v02r01.20240120.nc"
    with xr.open_dataset(path) as day:
        moved = day.load().assign_coords(lat=day.lat + 0.0001)
    os.remove(path)
    moved.to_netcdf(path)


def no_time(folder):
    path = f"{folder}/oisst-avhrr-v02r01.20240120.nc"
    with xr.open_dataset(path) as day:
        flat = day.load().isel(time=0, drop=True)
    os.remove(path)
    flat.to_netcdf(path)


attempt("empty glob", lambda: xr.open_mfdataset("nothing_here/*.nc"))
attempt("cropped file, by_coords", lambda: xr.open_mfdataset(sorted(glob.glob("cropped/*.nc"))))
attempt("cropped file, join='override'", lambda: xr.open_mfdataset(sorted(glob.glob("cropped/*.nc")), join="override"))
attempt("missing ice, nested", lambda: dict(xr.open_mfdataset(sorted(glob.glob("missing_variable/*.nc")),
                                                           combine="nested", concat_dim="time").sizes))
attempt("latitude shifted 0.0001°", lambda: xr.open_mfdataset(jan_plus(shifted, "shifted")))
attempt("HTML page as .nc", lambda: xr.open_mfdataset(["broken/oisst-avhrr-v02r01.20240201.nc"]))
attempt("truncated download", lambda: xr.open_mfdataset(["broken/oisst-avhrr-v02r01.20240202.nc"]))
attempt("one file without time", lambda: xr.open_mfdataset(jan_plus(no_time, "notime")))

mv = xr.open_mfdataset(sorted(glob.glob("missing_variable/*.nc")))
print(f"{'missing ice on 10 January':34} ok: ice all NaN that day: "
      f"{bool(mv.ice.sel(time='2024-01-10').isnull().all())}")
dup = sorted(glob.glob("clean/*.nc")) + ["broken/oisst-avhrr-v02r01.20240115_preliminary.nc"]
nested = xr.open_mfdataset(sorted(dup), combine="nested", concat_dim="time")
bycoords = xr.open_mfdataset(dup)
print(f"{'preliminary duplicate':34} nested {nested.sizes['time']} steps "
      f"({int(nested.indexes['time'].duplicated().sum())} duplicate), by_coords {bycoords.sizes['time']}")
shuffled = sorted(glob.glob("clean/*.nc"))
random.Random(1).shuffle(shuffled)
mixed = xr.open_mfdataset(shuffled, combine="nested", concat_dim="time")
attempt("shuffled list, nested, sliced", lambda: mixed.sel(time=slice("2024-01-05", "2024-01-09")).sizes)

year = sorted(glob.glob("oisst-2024/*.nc"))
t = time.perf_counter()
xr.open_mfdataset(year).close()
print(f"sequential open of {len(year)} files: {time.perf_counter() - t:.1f} s")
code = "import xarray as xr, glob; xr.open_mfdataset(sorted(glob.glob('oisst-2024/*.nc')), parallel=True).sst.mean().compute()"
for run in range(6):
    r = subprocess.run([sys.executable, "-W", "ignore", "-c", code], capture_output=True, text=True)
    last = (r.stderr.strip().splitlines() or ["(no message)"])[-1]
    print(f"parallel=True attempt {run + 1}: exit {r.returncode}: {last[:90]}")

empty glob                         OSError: no files to open
cropped file, by_coords            ValueError: Resulting object does not have monotonic global indexes along dimension lon
cropped file, join='override'      AlignmentError: cannot align objects with join='override' with matching indexes along dimension 'time' that don
missing ice, nested                ok: {'time': 31, 'zlev': 1, 'lat': 720, 'lon': 1440}
latitude shifted 0.0001°           ValueError: Resulting object does not have monotonic global indexes along dimension lat
HTML page as .nc                   ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'h5netcdf',
truncated download                 OSError: [Errno -101] NetCDF: HDF error: 'broken/oisst-avhrr-v02r01.20240202.nc'
one file without time              ValueError: Every dimension requires a corresponding 1D coordinate and index for inferring concatenation or
missing ice on 10 January          ok: ice all NaN that day: True
preliminary duplicate              nested 32 steps (1 duplicate), by_coords 31
shuffled list, nested, sliced      KeyError: 'Value based partial slicing on non-monotonic DatetimeIndexes with non-existing keys is not all
sequential open of 366 files: 1.5 s
parallel=True attempt 1: exit -11: RuntimeError: NetCDF: HDF error
parallel=True attempt 2: exit -11: RuntimeError: NetCDF: HDF error
parallel=True attempt 3: exit -11: (no message)
parallel=True attempt 4: exit -11: RuntimeError: NetCDF: HDF error
parallel=True attempt 5: exit -11: (no message)
parallel=True attempt 6: exit -11: (no message)

Errors that point at the wrong file

Why the errors point at the wrong thing

open_mfdataset does not report which file caused a problem. A cropped file makes the concatenated longitude index non-monotonic, so the error describes the index. A file no backend can read makes backend detection fail, so the error describes the backends. A process that cannot open more files makes the same detection fail. Scanning files individually turns each of those into a named file and a specific problem.

Why some problems raise nothing

xarray is designed to combine imperfect data: an outer join pads missing coordinates, a missing variable becomes NaN, nested concatenation keeps every step it is given. Those defaults keep workflows running, and they mean a duplicate or incomplete file changes the data without any message. Strict options — join="exact" and explicit checks after opening — trade that convenience for early failure.

Why the default join is changing

Opening several of these folders printed FutureWarning: In a future version of xarray the default value for join will change from join='outer' to join='exact'. When that change lands, mismatched grids will fail instead of being padded — the behaviour Example 2 opts into now.

Why parallel opening is unsafe here

Each file open enters the HDF5 C library. The builds used here are not compiled with thread safety, so concurrent opens from dask's thread pool corrupt shared state. The symptoms vary from run to run because they depend on timing, which is why a crash in one attempt and an HDF error in the next point at the same cause.

Hangs, huge collections and compressed files

• Hangs rather than errors usually mean a network file system or a remote URL; test with a local copy first.
• Hundreds of thousands of files make even the metadata scan slow; build a Kerchunk index or convert.
• Different chunking between files is allowed but produces uneven dask chunks.
• Mixed calendars across model files fail when times are combined; see fixing cftime errors.
• Engine-specific failures can be isolated by passing engine="netcdf4" explicitly.
• Compressed .nc.gz files are not NetCDF until decompressed.
• "double free or corruption" after the results print is not your files. Measured on 10 October 2026: with cfgrib's ecCodes 2.49 wheel and cartopy 0.26 in one environment, any script that imports xarray and opens a file prints its answer and then aborts at shutdown with exit code 134, because xarray loads every backend plugin and the two libraries' C++ runtimes collide. The output is complete; a fresh environment without one of the two exits cleanly.
• Upcoming default changes to join, compat and data_vars will turn some silent cases into errors; set them explicitly to keep behaviour stable.

Internal links

• How to open hundreds of NetCDF files as one dataset — the working workflow
• Fixing xarray's "did not find a match in any of xarray's installed backends" — the backend error in detail
• How to open a NetCDF file in Python with xarray — checking single files
• Fixing cftime and out-of-bounds datetime errors in xarray — calendar mismatches across files
• Regridding explained: bilinear, conservative and nearest for gridded data — putting files on one grid
• GRIB, NetCDF or Zarr: choosing a format for gridded data — converting a collection
• Lazy loading explained: why xarray reads nothing until you ask — what happens after opening
• NetCDF and gridded data explained: dimensions, variables and attributes — what the scan is checking

FAQ

Why does open_mfdataset say the index is not monotonic?

One file has a different or shifted grid. A single cropped OISST file raised the error for longitude; a latitude shifted by 0.0001° raised it for latitude. Scan the files' grids to find it.

Why does open_mfdataset say no backend matches when netCDF4 is installed?

One file is not NetCDF — often an HTML error page from a failed download — or the process cannot open more files. Both produced the same message here.

Why does my combined dataset have duplicate dates?

A duplicate or preliminary file is in the folder. Nested combining kept both copies of the day; check index.has_duplicates after opening.

How do I fix "too many open files" in open_mfdataset?

Lower xarray's file cache with xr.set_options(file_cache_maxsize=...) or raise the operating system limit. With a 64-file limit, a cache of 32 let the 366-file year open.

Why does parallel=True crash open_mfdataset?

The HDF5 library underneath netCDF4 is not safe for concurrent use in threads in these builds. Six attempts crashed; open sequentially or use separate processes.

How do I make open_mfdataset fail loudly on mismatched files?

Pass join="exact" and check the time index for order and duplicates after opening. The cropped file then raised an alignment error instead of being padded.