🔧 Fix Sea surface temperature & marine heatwaves Files & formats Notebook By · Updated

Fixing SST Values That Read as Thousands Instead of Degrees

On this page

Problem statement

Sea surface temperatures read from a NetCDF file come out as 2,680 instead of 26.8 °C, land cells are -999 instead of missing, and a mean over the ocean is negative. The file is not corrupt. Many products, including NOAA's daily OISST files from NCEI, store temperatures as 16-bit integers with a scale_factor of 0.01 and a _FillValue of -999, which halves the file size, and rely on the reader to unpack them. xarray and netCDF4 do that by default; reading with the unpacking turned off, with h5py, or with a tool that ignores the attributes does not.

This page reads one OISST file packed and unpacked, decodes the integers by hand, shows the precision packing costs, and writes a packed file back correctly.

Quick answer

Let the library apply scale_factor, add_offset and _FillValue, or apply them yourself:

raw = xr.open_dataset(P, mask_and_scale=False).sst.squeeze(drop=True)
dec = xr.open_dataset(P).sst.squeeze(drop=True)
a = raw.attrs
print(f"stored: {raw.dtype}, scale_factor {a['scale_factor']}, add_offset {a['add_offset']}, _FillValue {a['_FillValue']}")
v = raw.sel(lat=0.125, lon=180.125)
print(f"one cell: stored {int(v)}, decoded {float(dec.sel(lat=0.125, lon=180.125)):.2f} °C, by hand {int(v) * a['scale_factor'] + a['add_offset']:.2f} °C")
stored: int16, scale_factor 0.009999999776482582, add_offset 0.0, _FillValue -999
one cell: stored 2972, decoded 29.72 °C, by hand 29.72 °C

Temperatures are stored as 16-bit integers. The scale factor reads as 0.009999999776482582, the nearest 32-bit float to 0.01. A cell on the equator stores 2972, which decodes to 29.72 °C, the same by hand as by xarray.

How a packed OISST value is stored and decoded. one cell, from disk to degreesstored int162972mask _FillValue first-999 becomes missingx scale_factor + add_offset2972 x 0.01 + 0 = 29.72decoded29.72 °C, as xarray reads itScale before masking and land becomes -9.99 °C spatialworkflow.io
From disk to degrees.

Undecoded means, decoding by hand, kept precision

1. What an undecoded mean looks like

Average the stored integers and the decoded values over the whole grid:

print(f"stored integers: mean {float(raw.mean()):,.1f}, min {int(raw.min())}, max {int(raw.max())}")
print(f"decoded: mean {float(dec.mean()):.2f} °C, min {float(dec.min()):.2f}, max {float(dec.max()):.2f} °C; missing {int(dec.isnull().sum()):,} cells")
print(f"cells equal to the fill value -999: {int((raw == a['_FillValue']).sum()):,}")
stored integers: mean 603.3, min -999, max 3378
decoded: mean 14.05 °C, min -1.80, max 33.78 °C; missing 345,650 cells
cells equal to the fill value -999: 345,650

Averaged as stored, the grid's mean is 603.3 and its minimum -999. Decoded, the mean is 14.05 °C, from -1.80 °C under sea ice to 33.78 °C, and the 345,650 cells holding the fill value are missing.

2. Decode by hand

Some readers return only the integers. Mask the fill value first, then scale; scaling first turns -999 into -9.99 °C, a plausible-looking temperature:

wrong = raw * a["scale_factor"] + a["add_offset"]
right = raw.where(raw != a["_FillValue"]) * a["scale_factor"] + a["add_offset"]
print(f"scale only: min {float(wrong.min()):.2f} °C, mean {float(wrong.mean()):.2f} °C")
print(f"mask, then scale: min {float(right.min()):.2f} °C, mean {float(right.mean()):.2f} °C; equal to xarray's decoding: {bool(np.allclose(right, dec, equal_nan=True))}")
scale only: min -9.99 °C, mean 6.03 °C
mask, then scale: min -1.80 °C, mean 14.05 °C; equal to xarray's decoding: True

Scaling without masking turns every land cell into -9.99 °C, which drags the mean down to 6.03 °C and looks like cold water, not missing data. Masking first gives exactly xarray's decoding.

Two world maps of the same OISST file: read without unpacking, values run from minus 999 on land to about 3,400 in the warmest water; read with unpacking, land is missing and the ocean runs from about minus 2 to 32 degrees. Two world maps of the same OISST file: read without unpacking, values run from minus 999 on land to about 3,400 in the warmest water; read with unpacking, land is missing and the ocean runs from about minus 2 to 32 degrees.
Packed and decoded.

3. The precision packing keeps

With a scale factor of 0.01, every value should be a whole number of hundredths of a degree. Check, and note that the stored scale factor is a 32-bit float:

hund = (dec * 100).round(6)
print(f"values that are whole hundredths: {float((hund == hund.round()).sum() / hund.notnull().sum()):.0%}")
print(f"distinct values in the grid: {np.unique(dec.values[~np.isnan(dec.values)]).size:,}; valid range {a['valid_min'] * a['scale_factor']:.2f} to {a['valid_max'] * a['scale_factor']:.2f} °C")
values that are whole hundredths: 67%
distinct values in the grid: 3,510; valid range -3.00 to 45.00 °C

Only 67% of decoded values are exact hundredths in floating point, because the scale factor itself is not exactly 0.01; the 3,510 distinct values are the hundredths between the coldest and warmest cells. The packing's valid range is -3.00 to 45.00 °C.

Writing a packed file and reading it back

The data used on this page

The NOAA OISST v2.1 daily file for 21 June 2023, oisst-avhrr-v02r01.20230621.nc, from NCEI's data archive.

from pathlib import Path

import numpy as np
import requests
import xarray as xr

Path("data").mkdir(exist_ok=True)
P = Path("data/oisst-avhrr-v02r01.20230621.nc")
if not P.exists():
    P.write_bytes(requests.get("https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/"
                               "202306/oisst-avhrr-v02r01.20230621.nc", timeout=300).content)
print(f"{P.name}: {P.stat().st_size / 1e6:.2f} MB")
oisst-avhrr-v02r01.20230621.nc: 1.52 MB

The file is 1.52 MB.

Example 1: write a packed file and read it back

To save space when writing, give xarray the packing in encoding; values beyond the precision of the scale factor are rounded:

enc = {"sst": {"dtype": "int16", "scale_factor": 0.01, "add_offset": 0.0, "_FillValue": -999, "zlib": True}}
dec.to_dataset(name="sst").to_netcdf("data/sst_packed.nc", encoding=enc)
dec.astype("float32").to_dataset(name="sst").to_netcdf("data/sst_float.nc", encoding={"sst": {"zlib": True}})
back = xr.open_dataset("data/sst_packed.nc").sst
print(f"packed file {Path('data/sst_packed.nc').stat().st_size / 1e6:.2f} MB, float32 file {Path('data/sst_float.nc').stat().st_size / 1e6:.2f} MB")
print(f"largest change on the round trip: {float(abs(back - dec).max()):.4f} °C")
packed file 0.68 MB, float32 file 1.24 MB
largest change on the round trip: 0.0000 °C

Packed, the file is 0.68 MB against 1.24 MB for compressed 32-bit floats, and reading it back changes no value by as much as 0.00005 °C, because the values were hundredths to begin with.

Global mean of one OISST day read three ways. mean of 21 June 2023 read three waysscaled, not masked6.03 °C (land at -9.99)masked, then scaled14.05 °Craw integers603.3 (off the scale)051015345,650 land cells hold the fill value -999 spatialworkflow.io
Three readings of one day.

Packing in the CF conventions

Packed NetCDF values: problems and fixes. packed value trapsTrapInsteadmask_and_scale=Falselet xarray decodescale before maskingmask -999 firstold encoding reused on writeset or drop encodingfiles with different scalesdecode, then concatenateOISST daily files from NCEI: int16, scale 0.01 spatialworkflow.io
Packed value traps.

Packing is part of the CF conventions for NetCDF. A variable's stored integer x represents the value x × scale_factor + add_offset, and _FillValue marks cells with no data; valid_min and valid_max bound the stored integers. A 16-bit integer with a scale of 0.01 covers -327.68 to 327.67 at a resolution of a hundredth, which is more than enough for sea surface temperature and needs half the space of a 32-bit float before compression.

Readers that follow the conventions unpack on reading, so most users never see the integers. They appear when the unpacking is switched off (mask_and_scale=False in xarray, set_auto_maskandscale(False) in netCDF4), when a file is read with a generic HDF5 library such as h5py, or when values are copied through a tool that drops attributes. The order matters if you decode yourself: mask the fill value, then scale, or -999 becomes a temperature of -9.99 °C.

Unsigned packing and re-encoding on write

Checks when reading packed NetCDF variables. before trusting numbers from a NetCDF filelook at the dtypeint16 means packedread scale_factor, add_offset0.01 and 0 for OISSTfind the _FillValue-999check the rangedecoded: -1.80 to 33.78 °Cdecide encoding on writepacked 0.68 MB vs 1.24 MBA mean in the hundreds is the first sign of packing spatialworkflow.io
Before trusting the numbers.
  • Unsigned packing: some products pack into unsigned bytes with _Unsigned = "true"; read with a library that honours it, or values above 127 come out negative.
  • Re-encoding on write: xarray keeps the source encoding on decoded variables and reuses it when writing, so arithmetic results can be silently re-packed and rounded; drop encoding or set it explicitly.
  • Different scales per file: products that change scale factors between files cannot be concatenated as raw integers; always decode first.
  • ERDDAP: requests in .nc format from ERDDAP usually return floats already unpacked; files from archives are where packing shows.

FAQ

Why are my NetCDF temperatures 100 times too large?

The variable is packed as integers with a scale_factor of 0.01 and was read without unpacking; let xarray decode it, or multiply by scale_factor after masking _FillValue.

What do scale_factor and add_offset mean?

The real value is the stored integer times scale_factor plus add_offset, as defined by the CF conventions.

Why is there -999 in my SST data?

It is the _FillValue for land and missing cells in OISST files; mask it before any arithmetic.

Does packing lose precision?

To the scale factor: OISST's 0.01 keeps hundredths of a degree, finer than the analysis error.