How to Handle Missing and Null Values in Spatial Datasets in Python

On this page

Spatial datasets are messy. Field surveys skip questions, sensors drop readings, joins fail to match, and legacy formats encode "no data" as magic numbers like -9999. Before you can map, aggregate, or model anything, you need to find those gaps and decide what to do about them.

This guide is about missing attribute values — the columns of data attached to your features (population, elevation, land-use code, sensor reading). It is not about null or empty geometries, which are a separate problem with separate tools; see the linked geometry guide for that. Here you will learn to detect missing attributes, recognise sentinel values that disguise themselves as real numbers, and apply the three core strategies: drop, fill, or flag.

Problem statement

You load a GeoDataFrame and the attribute columns are not clean. Typical symptoms:

Blank cells that import as None, NaN or empty strings ("").
"NA", "N/A", "null", "-" strings that pandas read literally as text instead of treating as missing.
Sentinel numbers like -9999, -999.9 or -32768 standing in for "no measurement" — these poison means, sums and colour scales because pandas treats them as real values.
Missing categories in classification columns (an empty code, a blank country or state).
Whole columns that are mostly or entirely empty.

All of these are attribute problems. The geometry of each row can be perfectly valid while its attributes are full of holes.

The examples on this page use a real case of all of them: the 17,088 Australian weather stations in NOAA's Global Historical Climatology Network (GHCN-Daily) station list. Its elevation column records "unknown" as -999.9 — for 3,785 stations — the state column is filled only for the United States, Canada, US territories and a handful of other countries, and most stations have no WMO number.

Quick answer

Read the file telling pandas which strings and numbers mean "missing", replace any remaining numeric sentinels with NaN, count the gaps, then choose to drop or fill. Never let a sentinel survive into a calculation.

For each gap, choose to drop, fill, or flag — based on how much is missing and how critical it is.

import geopandas as gpd
import numpy as np
import pandas as pd

# 1. Read, telling pandas what "missing" looks like in this file
raw = pd.read_fwf("data/ghcnd-stations.txt", colspecs=COLSPECS, names=NAMES, dtype={"wmo": str})
raw = raw[raw["id"].str.startswith("AS")]                          # Australia
print("mean elevation with the sentinel in:", round(raw["elev"].mean(), 1), "m")

# 2. Replace the numeric sentinel with NaN
raw["elev"] = raw["elev"].replace(-999.9, np.nan)
stations = gpd.GeoDataFrame(raw, geometry=gpd.points_from_xy(raw["lon"], raw["lat"]), crs="EPSG:4326")
print("mean elevation without it:           ", round(stations["elev"].mean(), 1), "m")

# 3. Count missing values per column
print(stations.isna().sum()[lambda s: s > 0].to_dict())

# 4a. Drop rows missing a required attribute...
located = stations.dropna(subset=["elev"])
# 4b. ...OR fill with a statistic and keep a flag
stations["elev_was_missing"] = stations["elev"].isna()
stations["elev_filled"] = stations["elev"].fillna(stations["elev"].median())
print(len(located), "stations with a known elevation;", int(stations["elev_was_missing"].sum()), "flagged and filled")

mean elevation with the sentinel in: -15.6 m
mean elevation without it:            264.5 m
{'elev': 3785, 'state': 17088, 'gsn': 17026, 'hcn_crn': 17088, 'wmo': 16291}
13303 stations with a known elevation; 3785 flagged and filled

COLSPECS and NAMES describe the file's fixed-width layout and are defined in The data used on this page. One sentinel value in a fifth of the rows moved the mean elevation of Australia's stations from 264.5 m to −15.6 m — below sea level — without an error or a warning.

Detect, then drop, fill or flag

Four of these are invisible to isna() — which is why counts look reassuring.

Detect missing values

Start by measuring the problem. isna() (identical to isnull()) returns a boolean mask; chain .sum() to count per column, and .mean() for a share:

missing_pct = (stations.isna().mean() * 100).round(1).sort_values(ascending=False)
print(missing_pct[missing_pct > 0].to_dict())
print(stations.loc[stations["elev"].isna(), ["id", "name", "elev"]].head(3).to_string(index=False))

{'hcn_crn': 100.0, 'state': 100.0, 'gsn': 99.6, 'wmo': 95.3, 'elev': 22.2}
         id                   name  elev
ASN00001011            PANTA DOWNS   NaN
ASN00001017            NULLA NULLA   NaN
ASN00001022 WYNDHAM SIX MILE HOTEL   NaN

isna() does not flag sentinel values or "NA" strings — they look like valid data until you convert them. Had the -999.9 not been replaced first, the elevation column would have reported no missing values at all.

Convert sentinel values to NaN

Sentinels are real-looking values that mean "no data". Find them by inspecting suspicious extremes and repeated values:

everything = pd.read_fwf("data/ghcnd-stations.txt", colspecs=COLSPECS, names=NAMES, dtype={"wmo": str})
raw_elev = everything.loc[everything["id"].str.startswith("AS"), "elev"]
print("countries with a state column:", sorted(everything.loc[everything["state"].notna(), "id"].str[:2].unique()))
print(raw_elev.describe()[["min", "25%", "50%", "max"]].to_dict())
print(raw_elev.value_counts().head(3).to_dict(), "| isna() before replacing:", raw_elev.isna().sum())

countries with a state column: ['AQ', 'BF', 'CA', 'CQ', 'FM', 'GQ', 'JQ', 'LQ', 'MQ', 'PS', 'RM', 'RQ', 'RS', 'US', 'VQ', 'WQ']
{'min': -999.9, '25%': 7.0, '50%': 120.0, 'max': 1957.0}
{-999.9: 3785, 300.0: 198, 20.0: 161} | isna() before replacing: 0

A minimum of −999.9 m on a continent whose lowest point is Kati Thanda–Lake Eyre, about 15 m below sea level, and one value repeated 3,785 times, are the signature. Only replace values that are genuinely invalid: a real 0 for rainfall is legitimate data, and so is a station a few metres below sea level. Know your dataset before you convert.

The same column read two ways: the sentinel hides every gap from isna() and drags the mean below sea level.

Decide drop vs fill vs flag

There is no universal rule. Pick based on how much is missing and whether the column is required:

Drop when a few rows lack a value you cannot reasonably estimate and that you need for the analysis.
Fill when the column is required downstream and a defensible estimate (constant, statistic, or spatial neighbour) is acceptable.
Flag when you want to keep every row and let downstream code or readers know which values were imputed.

In practice you often combine them: flag first, then fill.

n = len(stations)
for col in ["elev", "wmo", "state"]:
    missing = stations[col].isna().sum()
    print(f"{col:5}: {missing:>6,} missing ({missing / n:.1%})")

elev :  3,785 missing (22.2%)
wmo  : 16,291 missing (95.3%)
state: 17,088 missing (100.0%)

Elevation is missing for 22% of stations, which is worth filling carefully; state is empty for every Australian station, because GHCN fills it for only 16 country codes, Australia not among them — there is nothing to fill, and the column should simply be dropped.

Drop rows or columns

Use dropna with subset to remove only rows missing a specific required column. Dropping blindly across all columns can wipe out most of your data:

print("dropna() with no subset keeps", len(stations.dropna()), "of", len(stations), "stations")
print("dropna(subset=['elev']) keeps  ", len(stations.dropna(subset=["elev"])))

threshold = 0.9
mostly_empty = stations.columns[stations.isna().mean() > threshold]
trimmed = stations.drop(columns=mostly_empty)
print("columns over 90% empty:", list(mostly_empty))

dropna() with no subset keeps 0 of 17088 stations
dropna(subset=['elev']) keeps   13303
columns over 90% empty: ['state', 'gsn', 'hcn_crn', 'wmo']

A bare stations.dropna() drops every row that has a gap in any column, and here that is every row: the state column is empty for all of them.

Fill with a constant or statistic

fillna replaces gaps. Use a constant for categories and a statistic for numbers — the median rather than the mean for skewed values such as elevation. A group-wise statistic is usually better than a global one; with no region column in the file, a 1° grid cell serves:

stations["wmo"] = stations["wmo"].fillna("none")                       # a constant for a code
stations["cell"] = (stations["lat"] // 1).astype(int).astype(str) + "," + (stations["lon"] // 1).astype(int).astype(str)
by_cell = stations.groupby("cell")["elev"].transform("median")
stations["elev_cell_fill"] = stations["elev"].fillna(by_cell).fillna(stations["elev"].median())
print(stations[["elev", "elev_cell_fill"]].describe().loc[["count", "mean", "50%"]].round(1).to_string())

          elev  elev_cell_fill
count  13303.0         17088.0
mean     264.5           257.4
50%      198.0           197.5

The global median fallback covers grid cells in which no station has a known elevation.

For time-ordered data, carrying the last value forward (ffill) is a common fill — and one that should come with a limit. Melbourne's Regional Office recorded daily maximum temperatures from 1910 until it closed in 2015:

melbourne = pd.read_csv("data/ASN00086071.csv", usecols=["DATE", "TMAX"], low_memory=False)
melbourne = melbourne[melbourne["DATE"] >= "1910-01-01"].sort_values("DATE")
print(len(melbourne), "days;", melbourne["TMAX"].isna().sum(), "missing")
melbourne["tmax_c"] = melbourne["TMAX"].ffill(limit=2) / 10                # GHCN stores tenths of a degree
print("after ffill(limit=2):", melbourne["tmax_c"].isna().sum(), "missing")

38357 days; 1 missing
after ffill(limit=2): 0 missing

A single missing day in 38,357 is a well-kept record; limit=2 means that a longer outage in some other series would stay visibly empty instead of repeating one day's temperature for a month.

Measured on this page by hiding a fifth of the known elevations: the nearest neighbour is more than three times as accurate as the global median.

Spatial-aware fill from nearby features

For spatial data, nearby features are often the best estimate (Tobler's first law: near things are more alike). Use sjoin_nearest to borrow a value from the closest feature that has one — in a projected CRS, so that "nearest" means nearest on the ground:

albers = stations.to_crs(3577)                                        # GDA94 / Australian Albers, metres
has_val = albers[albers["elev"].notna()]
no_val = albers[albers["elev"].isna()]
filled = gpd.sjoin_nearest(no_val[["geometry"]], has_val[["elev", "geometry"]], how="left", distance_col="fill_dist")
filled = filled[~filled.index.duplicated()]                           # ties at the same distance
stations.loc[filled.index, "elev_nn"] = filled["elev"]
stations["elev_nn"] = stations["elev_nn"].fillna(stations["elev"])
print(f"{len(filled):,} filled; neighbour distance median {filled['fill_dist'].median() / 1000:.1f} km, "
      f"max {filled['fill_dist'].max() / 1000:.0f} km")

3,785 filled; neighbour distance median 6.3 km, max 96 km

sjoin_nearest returns one row per tied neighbour, so drop duplicate indices before writing back. And look at the distances: a neighbour 6 km away is a reasonable guess for elevation; the farthest here, 96 km away, is not.

Test the fill before trusting it

Every fill is a guess, and the only way to know how good a guess is, is to make it where you already know the answer. Hide a fifth of the known elevations, fill them each way, and compare:

known = albers[albers["elev"].notna()].copy()
hidden = np.random.default_rng(42).random(len(known)) < 0.2
test, train = known[hidden].copy(), known[~hidden].copy()
for frame in (test, train):
    frame["cell"] = (frame["lat"] // 1).astype(int).astype(str) + "," + (frame["lon"] // 1).astype(int).astype(str)

guesses = {
    "global median": np.full(len(test), train["elev"].median()),
    "1° cell median": test["cell"].map(train.groupby("cell")["elev"].median()).fillna(train["elev"].median()).to_numpy(),
}
nearest = gpd.sjoin_nearest(test[["geometry"]], train[["elev", "geometry"]], how="left")
guesses["nearest station"] = nearest[~nearest.index.duplicated()]["elev"].to_numpy()
for name, guess in guesses.items():
    error = np.abs(guess - test["elev"].to_numpy())
    print(f"{name:16} mean error {error.mean():6.1f} m   median error {np.median(error):6.1f} m")

global median    mean error  187.5 m   median error  140.2 m
1° cell median   mean error  103.8 m   median error   46.5 m
nearest station  mean error   51.1 m   median error   21.0 m

On Australia's stations the nearest neighbour is more than three times as accurate as the global median, and the grid-cell median sits in between. That ranking is a property of this data — elevation is strongly spatially autocorrelated — not a law; a quantity that varies more locally would narrow the gap.

Keep an audit flag

Before any fill, record which values were originally missing. A boolean column lets you exclude or weight imputed values later and keeps your cleaning honest:

print(stations["elev_was_missing"].sum(), "elevations imputed;",
      "mean over measured stations only:", round(stations.loc[~stations["elev_was_missing"], "elev"].mean(), 1), "m")

3785 elevations imputed; mean over measured stations only: 264.5 m

Filling gaps in Australia's weather-station list

The data used on this page

Stations — NOAA's GHCN-Daily station list, ghcnd-stations.txt, a fixed-width text file of every station in the network with its latitude, longitude, elevation, state, name and WMO number. The first two letters of the ID are the country (FIPS code), and AS is Australia. NOAA updates the list as stations join.
One station's record — the daily file for Melbourne Regional Office (ASN00086071), 1855–2015.

import os

import geopandas as gpd
import numpy as np
import pandas as pd
import requests

os.makedirs("data", exist_ok=True)
for name, url in [("ghcnd-stations.txt", "https://www.ncei.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt"),
                  ("ASN00086071.csv", "https://www.ncei.noaa.gov/data/global-historical-climatology-network-daily/"
                                      "access/ASN00086071.csv")]:
    if not os.path.exists(f"data/{name}"):
        reply = requests.get(url, timeout=300)
        reply.raise_for_status()
        open(f"data/{name}", "wb").write(reply.content)

# the column layout from NOAA's ghcnd readme
COLSPECS = [(0, 11), (12, 20), (21, 30), (31, 37), (38, 40), (41, 71), (72, 75), (76, 79), (80, 85)]
NAMES = ["id", "lat", "lon", "elev", "state", "name", "gsn", "hcn_crn", "wmo"]
print(sum(1 for _ in open("data/ghcnd-stations.txt")), "stations in the network")

133039 stations in the network

Example 1: Read with the sentinel declared up front

declared = pd.read_fwf("data/ghcnd-stations.txt", colspecs=COLSPECS, names=NAMES, dtype={"wmo": str},
                       na_values={"elev": ["-999.9"], "name": ["", "NA", "N/A"]})
declared = declared[declared["id"].str.startswith("AS")]
print(declared["elev"].isna().sum(), "elevations missing on read; min", declared["elev"].min(), "m")

3785 elevations missing on read; min 0.0 m

Declaring the sentinel in na_values, per column, means no later step ever sees it. The same argument works for read_csv.

Example 2: A reusable missing-value report

def missing_report(gdf):
    return (pd.DataFrame({"missing": gdf.isna().sum(), "pct": (gdf.isna().mean() * 100).round(1),
                          "dtype": gdf.dtypes.astype(str)})
            .query("missing > 0").sort_values("pct", ascending=False))


print(missing_report(gpd.GeoDataFrame(declared, geometry=gpd.points_from_xy(declared["lon"], declared["lat"]))))

         missing    pct    dtype
state      17088  100.0      str
hcn_crn    17088  100.0      str
gsn        17026   99.6      str
wmo        16291   95.3      str
elev        3785   22.2  float64

Example 3: Detect likely sentinels automatically

def find_sentinels(series, candidates=(-9999, -999.9, -999, 9999, -32768, 32767)):
    return {c: int((series == c).sum()) for c in candidates if (series == c).any()}


undeclared = pd.read_fwf("data/ghcnd-stations.txt", colspecs=COLSPECS, names=NAMES, dtype={"wmo": str})
for col in undeclared.select_dtypes("number").columns:
    hits = find_sentinels(undeclared[col])
    if hits:
        print(col, hits)

elev {-999.9: 4737}

Run on the whole network, the check finds the elevation sentinel in stations far beyond Australia; latitude and longitude are clean.

Example 4: Nearest-neighbour fill with a distance cap

cap_m = 5_000
capped = filled[filled["fill_dist"] <= cap_m]
print(f"{len(capped):,} of {len(filled):,} missing elevations have a measured station within {cap_m / 1000:.0f} km;",
      f"the other {len(filled) - len(capped):,} stay NaN")

1,535 of 3,785 missing elevations have a measured station within 5 km; the other 2,250 stay NaN

A cap turns "nearest" into "near enough". Rows beyond it keep their gap, and their flag, rather than inheriting the elevation of a station on the other side of a range.

NaN, None and sentinel values

NaN vs None vs sentinel. NaN is a special floating-point value (numpy.nan) that pandas uses to mark missing numeric data; arithmetic on it propagates (NaN + 5 == NaN), and isna() detects it. None is Python's null object, common in object/string columns; pandas also treats it as missing. A sentinel is an ordinary value (-9999) that a dataset conventionally uses to mean "missing" — but pandas has no way to know that, so it treats -9999 as a real number. The whole point of the sentinel-conversion step is to turn those into genuine NaN so the missing-data machinery can see them.

Filling is the only one of the three that can change a result without a trace.

Why dropping can bias analysis. Missing data is rarely random in the real world. Sensors fail more often in harsh, remote, or extreme locations; surveys are skipped in hard-to-reach areas. If you dropna those rows, you systematically remove a particular kind of place, and your remaining sample no longer represents the study area. A national average computed after dropping all the remote stations will be skewed toward the easy-to-measure ones.

Why a "was_missing" flag matters. Once you fill a value, it becomes indistinguishable from a real measurement — that information loss is permanent unless you record it. A boolean *_was_missing column lets you audit how much of a result rests on imputed data, exclude imputed rows from sensitive statistics, or even use "missingness" itself as a predictor. It costs one column and buys you transparency and reproducibility.

When a fill distorts the data

The techniques in this article handle the left-hand column.

Missing attributes vs null geometries are different problems

A row can have a perfectly valid point and a missing population, or a valid population and a None geometry. The techniques here (isna, fillna, dropna on attribute columns) handle the attribute case. Null or empty geometries need geometry-specific checks like gdf.geometry.isna() and gdf.geometry.is_empty — see the linked geometry guide. Don't conflate the two.

Sentinel values vary by dataset

-9999 is common but far from universal. You'll meet 9999, -32768/32767 (signed 16-bit limits), -1, 255, 1e20, and 0. Always read the dataset's metadata or data dictionary for its declared "NoData" value rather than guessing, and confirm with describe() and value_counts().

Mean-fill distorts distributions

Filling every gap with the column mean (or median) collapses variance: a spike appears at the fill value and the spread shrinks. This can mislead later statistics, regressions, and choropleth class breaks. For a handful of values it's fine; for a large fraction, prefer a model-based or spatial estimate and always keep the audit flag.

Spatial autocorrelation makes nearest-neighbour fill reasonable

Because nearby locations tend to have similar values, borrowing from the closest feature is often more accurate than a global statistic for spatial attributes. This works best where the variable is genuinely smooth in space (elevation, temperature) and worse where it jumps sharply across boundaries (administrative codes, land ownership). Cap the fill distance so you don't borrow from features that are too far away to be relevant.

Document every fill

Record what you did: which columns were filled, with what method, how many values, and the date. Store it in a sidecar note, a commit message, or the *_was_missing flags themselves. Future-you (and reviewers) need to know which numbers were measured and which were invented.

dtype changes after fillna

Watch your dtypes. A clean integer column that contains NaN is promoted to float64, and filling a numeric column with a string ("unknown") turns the whole column into object. After filling, check gdf.dtypes and cast back if needed (gdf["count"] = gdf["count"].astype("Int64") — pandas' nullable integer type tolerates NA).

Internal links

FAQ

What's the difference between a missing attribute and a null geometry?

A missing attribute is a gap in a data column — no recorded population, elevation, or land-use code for a feature whose location is known. A null geometry is a row whose shape is None or empty, regardless of its attributes. They're detected and fixed differently: attribute gaps with isna/fillna/dropna on the column, geometry nulls with geometry.isna()/is_empty.

How do I find missing values that are hidden as -9999?

Plain isna() won't catch them because -9999 is a valid number to pandas. Inspect each numeric column with describe() and value_counts() to spot implausible extremes or oddly frequent values, then replace([-9999, 9999], np.nan) to convert the real sentinels into genuine NaN.

Should I drop or fill missing data?

Drop when only a few rows lack a value you truly need and can't estimate; fill when the column is required downstream and a defensible estimate exists. If in doubt, flag-and-fill rather than drop, because dropping non-random gaps can bias your results. The right choice depends on how much is missing and how the data will be used.

What does na_values do when reading a file?

na_values tells the reader which strings (or numbers) to treat as missing at load time, so "NA", "null", "-", or "-9999" become NaN immediately instead of being read as literal text or numbers. It's available on pd.read_csv; for vector formats read with GeoPandas, convert sentinels with replace after loading.

Is mean or median better for filling?

Median for skewed data (most environmental variables) and when outliers are present, because it isn't dragged toward extremes; mean for roughly symmetric distributions. Better still, use a group-wise or spatial estimate so the fill reflects local conditions. Whichever you pick, set a *_was_missing flag so the imputation is auditable.

Why did my integer column turn into floats after handling missing values?

NaN is a float, so any integer column containing it is automatically promoted to float64. To keep integer semantics with missing values, use pandas' nullable integer type: gdf["count"] = gdf["count"].astype("Int64") (capital I), which tolerates pd.NA.

How do I fill a value from the nearest feature that has one?

Split the frame into rows that have the value and rows that don't, reproject to a metric CRS, then gpd.sjoin_nearest the missing rows against the populated ones to borrow the closest value. Use distance_col to capture how far each match was, and cap that distance so you don't import values from features that are too far away.

Can I just use fillna(0) for everything?

Rarely a good idea. 0 is a real, meaningful value for most measurements (zero rainfall, zero elevation at sea level), so filling with it silently injects false data and skews sums, means, and maps. Use 0 only when zero is genuinely the correct default, and otherwise prefer a statistic, a spatial estimate, or an explicit "unknown" category — always with an audit flag.