🔧 Fix Gridded & climate data Data cleaning Gridded & Climate Data · step 20 of 20 Updated

Fixing cftime and Out-of-Bounds Datetime Errors in xarray

Problem statement

Climate model output rarely uses the calendar pandas knows. A model year may have 365 days (noleap) or 360 (360_day), and scenario runs extend past 2262, the last date a nanosecond timestamp can hold. xarray decodes such times into cftime objects instead of NumPy datetimes, and ordinary date code then fails: selections raise, plots refuse, two models subtract to an empty array, conversions overflow.

Measured on three CMIP6 monthly near-surface temperature datasets read from the public Pangeo Zarr store — UKESM1-0-LL (360_day), CESM2 (noleap) and a CNRM-ESM2-1 scenario ending in 2299:

  • Selecting 2000-01-01 to 2000-12-31 from UKESM raised ValueError: invalid day number, because 31 December does not exist in a 360-day year. Selecting "2000-01" to "2000-12" returned 12 months.
  • UKESM minus CESM returned an array of size 0 — no error — because their mid-month timestamps never coincide. Both resampled to month starts and converted to the standard calendar, the difference had all 1,980 months.
  • np.datetime64("2300-01-01", "ns") silently became 1715-06-13, while pandas raised OutOfBoundsDatetime for the same date.
  • The CNRM run's times converted cleanly with time_unit="s", and failed at 2262-04-16 with the default nanoseconds.

Quick answer

Read the calendar, select with partial date strings, and convert calendars explicitly before combining models:

import xarray as xr

UK = "https://storage.googleapis.com/cmip6/CMIP6/CMIP/MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/v20190406/"
tas = xr.open_zarr(UK)["tas"].sel(lat=51.5, lon=0, method="nearest")
print(tas.time.dt.calendar, type(tas.indexes["time"]).__name__)

year_2000 = tas.sel(time=slice("2000-01", "2000-12"))                      # partial strings work in any calendar
standard = tas.resample(time="MS").mean().convert_calendar("standard", align_on="date")
360_day CFTimeIndex

For dates after 2262, decode with decode_times=xr.coders.CFDatetimeCoder(time_unit="s") to get NumPy datetimes with second resolution instead of cftime objects.

Triage of calendar errors: invalid day number, KeyError on a pandas timestamp, empty result when subtracting models, plotting failure, and out-of-bounds nanosecond timestamps, with the fix for each.
Every one of these is the same fact — the dates are not standard-calendar nanosecond timestamps — surfacing in a different function.

Step-by-step solution

1. Check the calendar and the index type

da.time.dt.calendar returns the calendar name — 360_day for UKESM, noleap for CESM2, and standard for the CNRM scenario, whose files say gregorian — and type(da.indexes["time"]) shows whether xarray built a CFTimeIndex or a pandas DatetimeIndex. UKESM's first timestamps were 16 January, 16 February and 16 March 1850; CESM2's were 15 January 12:00, 14 February 00:00 and 15 March 12:00.

2. Select with dates that exist in that calendar

A cftime index compares against cftime dates, not pandas timestamps. On UKESM, sel(time=pd.Timestamp("2000-06-16")) and a numpy.datetime64 both raised KeyError; a slice of pandas timestamps raised TypeError: cannot compare cftime.Datetime360Day(...) and Timestamp(...) (different calendars). Strings work, provided the date exists: "2000-12-31" failed in the 360-day calendar and "2000-02-29" failed in noleap. Year-month strings exist in every calendar and are the safest choice.

3. Do not subtract or align models on raw timestamps

xarray aligns on labels. UKESM's 16th-of-the-month stamps and CESM2's mid-month stamps had no timestamps in common, so ukesm - cesm returned an empty array. xr.align(ukesm, cesm, join="exact") at least raised AlignmentError. Resampling both to month starts is not enough either: the labels were still in two different calendars and the difference was still empty (Example 2).

4. Convert calendars explicitly

convert_calendar("standard") worked directly on the noleap data. On 360_day data it raised ValueError: Argument align_on must be specified with either 'date' or 'year'. align_on="date" keeps dates that exist in both calendars and drops the rest, which suits monthly means labelled at month starts; align_on="year" spreads the 360 days over the year by position. After resampling to month starts and converting, UKESM and CESM2 subtracted cleanly over all 1,980 months.

5. Weight by month length when averaging model months

A noleap February has 28 days, a 360_day February 30. Annual means from monthly values should weight by time.dt.days_in_month in the model's own calendar. For CESM2 at London the unweighted and weighted annual means differed by up to 0.058 K; for UKESM, where every month has 30 days, by 10⁻⁵ K.

6. Handle dates beyond 2262

NumPy's nanosecond datetime64 covers 1678 to 2262. The CNRM scenario runs to December 2299, so xarray kept cftime objects and warned that it could not decode to datetime64[ns]. CFTimeIndex.to_datetimeindex() then failed with Cannot convert date 2262-04-16 ... without overflow; with time_unit="s" it returned a DatetimeIndex ending on 16 December 2299. Opening with CFDatetimeCoder(time_unit="s") produced datetime64[s] times directly (Example 3).

7. Be suspicious of nanosecond conversions you did not ask for

pandas 3 creates Timestamp("2300-01-01") at second resolution without complaint, and raises OutOfBoundsDatetime when asked for nanoseconds. NumPy does not raise: np.datetime64("2300-01-01", "ns") wrapped round to 13 June 1715. Anything that forces nanoseconds on late dates — an explicit unit="ns", an old library, a hand-written conversion — can corrupt times silently.

8. Plot and export with the right tools

da.plot() on a cftime axis raised ImportError: ... requires the optional nc-time-axis (v1.2.0 or later) package, and plain matplotlib failed converting the dates to floats. Writing Parquet from a cftime index raised ArrowInvalid: Could not convert cftime.DatetimeGregorian. Install nc-time-axis for plotting, and convert calendars or times before exporting to formats that expect standard timestamps.

Bar chart of the number of months in the difference between UKESM and CESM2 series: zero on raw timestamps, zero after resampling to month starts, and 1,980 after converting both to the standard calendar.
Two models' monthly series only line up once both the labels and the calendar agree.

Code examples

Example 1 — report the calendar and select safely 

import xarray as xr

STORE = "https://storage.googleapis.com/cmip6/CMIP6/CMIP/"
MODELS = {
    "UKESM1-0-LL": STORE + "MOHC/UKESM1-0-LL/historical/r1i1p1f2/Amon/tas/gn/v20190406/",
    "CESM2": STORE + "NCAR/CESM2/historical/r1i1p1f1/Amon/tas/gn/v20190308/",
}

series = {}
for name, url in MODELS.items():
    da = xr.open_zarr(url)["tas"].sel(lat=51.5, lon=0, method="nearest").load()
    series[name] = da
    first = [str(t)[:16] for t in da.time.values[:2]]
    print(f"{name}: calendar {da.time.dt.calendar}, index {type(da.indexes['time']).__name__}, "
          f"{da.sizes['time']} steps from {first}")
    try:
        da.sel(time=slice("2000-01-01", "2000-12-31"))
        print("  full dates: ok")
    except ValueError as error:
        print(f"  full dates: {error}")
    print(f"  year-month strings: {da.sel(time=slice('2000-01', '2000-12')).sizes['time']} months")
UKESM1-0-LL: calendar 360_day, index CFTimeIndex, 1980 steps from ['1850-01-16 00:00', '1850-02-16 00:00']
  full dates: invalid day number provided in cftime.Datetime360Day(2000, 12, 31, 0, 0, 0, 0, has_year_zero=True)
  year-month strings: 12 months
CESM2: calendar noleap, index CFTimeIndex, 1980 steps from ['1850-01-15 12:00', '1850-02-14 00:00']
  full dates: ok
  year-month strings: 12 months

Example 2 — put two models on one calendar 

def to_standard_months(da):
    months = da.resample(time="MS").mean()
    align = "date" if da.time.dt.calendar == "360_day" else None
    return months.convert_calendar("standard", align_on=align)


uk, ce = series["UKESM1-0-LL"], series["CESM2"]
print("raw timestamps:        ", (uk - ce).sizes["time"], "months")
print("resampled only:        ", (uk.resample(time="MS").mean() - ce.resample(time="MS").mean()).sizes["time"], "months")
print("resampled and converted:", (to_standard_months(uk) - to_standard_months(ce)).sizes["time"], "months")
raw timestamps:         0 months
resampled only:         0 months
resampled and converted: 1980 months

Example 3 — decode dates beyond 2262 

import numpy as np
import pandas as pd

CNRM = "https://storage.googleapis.com/cmip6/CMIP6/ScenarioMIP/CNRM-CERFACS/CNRM-ESM2-1/ssp534-over/r1i1p1f2/Amon/tas/gr/v20190328/"

default = xr.open_zarr(CNRM)
print("default:", type(default.indexes["time"]).__name__, default.time.values[-1])
for unit in ("ns", "s"):
    try:
        print(f"to_datetimeindex(time_unit={unit!r}):", default.indexes["time"].to_datetimeindex(time_unit=unit)[-1])
    except ValueError as error:
        print(f"to_datetimeindex(time_unit={unit!r}): {str(error)[:60]}...")

seconds = xr.open_zarr(CNRM, decode_times=xr.coders.CFDatetimeCoder(time_unit="s"))
print("decoded to seconds:", seconds.time.dtype, seconds.time.values[-1])
print("numpy ns, no error:", np.datetime64("2300-01-01", "ns"))
try:
    pd.Timestamp("2300-01-01").as_unit("ns")
except pd.errors.OutOfBoundsDatetime as error:
    print("pandas ns:", error)
default: CFTimeIndex 2299-12-16 12:00:00
to_datetimeindex(time_unit='ns'): Cannot convert date 2262-04-16 00:00:00 to a date in the sta...
to_datetimeindex(time_unit='s'): 2299-12-16 12:00:00
decoded to seconds: datetime64[s] 2299-12-16T12:00:00
numpy ns, no error: 1715-06-13T00:25:26.290448384
pandas ns: Cannot cast 2300-01-01 00:00:00 to unit='ns' without overflow.

The CNRM store is a scenario run, so its calendar is the ordinary Gregorian one; only the range, not the calendar, stops nanosecond timestamps.

Explanation

Why models use non-standard calendars

A climate model does not need leap years or months of unequal length; it needs a year of fixed length so that the seasonal cycle of the model's sun is identical every year. A 360-day calendar makes every month 30 days; a noleap calendar keeps the real month lengths but drops 29 February. Those dates cannot be represented by pandas, which implements only the proleptic Gregorian calendar.

Why the errors appear in unrelated places

A CFTimeIndex supports most of what xarray does itself — partial-string selection, resample, groupby("time.month"), dt accessors, writing NetCDF. Anything that hands the times to pandas, NumPy or matplotlib — timestamp objects, interp against a DatetimeIndex, Parquet, plotting — meets objects it cannot convert. The error message names whatever function did the converting.

Why "date" and "year" alignment differ

Converting 360-day dates to the standard calendar has no single right answer. align_on="date" maps each date to the same month and day and drops dates such as 30 February; align_on="year" maps position in the year, so the 180th of 360 days becomes about the 183rd of 365. For monthly means resampled to month starts, "date" keeps every month; for daily data, "year" keeps every day.

Why 2262 is a limit at all

A signed 64-bit count of nanoseconds since 1970 runs out on 11 April 2262. pandas 2 and 3 support coarser units that extend the range enormously, and xarray can decode to them, but nanoseconds remain the default in much code. Second resolution is far finer than monthly or daily climate data needs.

Table comparing how NumPy, pandas and xarray handled a date after 2262 at nanosecond and second resolution.
The only silent failure was NumPy's: a nanosecond datetime64 for 2300 wrapped round to 1715.

Edge cases or notes

  • use_cftime is deprecated as a keyword; pass decode_times=xr.coders.CFDatetimeCoder(use_cftime=True) instead.
  • decode_times=False exposes the raw numbers, units and calendar attribute when decoding fails.
  • xr.date_range(..., calendar="360_day", use_cftime=True) builds matching cftime indexes for comparisons.
  • Time differences on a cftime axis are ordinary timedeltas: 30 days between UKESM months.
  • polyfit along time uses nanoseconds as the unit, so slopes per time step look vanishingly small.
  • Resampling speed was similar either way: 0.061 s for annual means of UKESM's 219 MB field with cftime, 0.058 s after conversion.
  • Observations are standard-calendar; convert model output before comparing with them.

FAQ

Why does xarray give me cftime objects instead of datetimes?

The file uses a calendar pandas does not support, such as noleap or 360_day, or dates outside 1678–2262. xarray keeps those times as cftime objects in a CFTimeIndex.

How do I select dates in a 360-day calendar?

Use year-month strings such as slice("2000-01", "2000-12"). Full dates must exist in the calendar: 31 December raised an invalid day number error for UKESM.

Why is the difference between two climate models empty?

Their timestamps do not match, so alignment keeps nothing. Resample both to month starts and convert both to the standard calendar; UKESM minus CESM2 then kept 1,980 months.

How do I convert a 360_day calendar to a standard one?

Use convert_calendar("standard", align_on="date") or align_on="year". Without align_on, xarray raises a ValueError for 360-day data.

How do I fix "Cannot convert date 2262-04-16 without overflow"?

Use second resolution: to_datetimeindex(time_unit="s"), or open with decode_times=xr.coders.CFDatetimeCoder(time_unit="s"). The CNRM run then decoded to December 2299.

How do I plot data with a cftime axis?

Install nc-time-axis, which xarray uses to plot cftime dates, or convert the calendar to standard first.