How to thin occurrence records spatially
On this page
Problem statement
Occurrence records cluster where people record, so a model fitted on raw presences learns the road network. Spatial thinning is the blunt, standard correction: keep at most one record per cell, or enforce a minimum distance between kept records, and accept the loss.
The question is not whether to thin but how far, and what it costs. On 6,332 Canada lynx records, a 10 km grid keeps 1,957; a 50 km grid keeps 832.
Quick answer
Grid thinning is a groupby, and it is usually enough:
import numpy as np, pandas as pd, geopandas as gpd
from gbif import search
lynx = search(taxonKey=2435263, hasCoordinate="true")
pts = gpd.GeoDataFrame(lynx, crs=4326,
geometry=gpd.points_from_xy(lynx.decimalLongitude,
lynx.decimalLatitude)).to_crs(3978)
def thin_by_grid(frame, cell_m, seed=0):
"""One record per cell, chosen at random within the cell."""
gx = np.floor(frame.geometry.x / cell_m).astype(int)
gy = np.floor(frame.geometry.y / cell_m).astype(int)
order = np.random.default_rng(seed).permutation(len(frame))
shuffled = frame.iloc[order].assign(_gx=gx.iloc[order].values, _gy=gy.iloc[order].values)
return shuffled.drop_duplicates(subset=["_gx", "_gy"]).drop(columns=["_gx", "_gy"])
print(f"{len(pts):,} records")
for cell in (1_000, 5_000, 10_000, 25_000, 50_000):
kept = thin_by_grid(pts, cell)
print(f" {cell // 1000:>2} km grid: {len(kept):>5,} kept "
f"({len(kept) / len(pts):>5.1%})")6,332 records
1 km grid: 2,549 kept (40.3%)
5 km grid: 2,246 kept (35.5%)
10 km grid: 1,957 kept (30.9%)
25 km grid: 1,321 kept (20.9%)
50 km grid: 832 kept (13.1%)
Every one of those is a defensible dataset. Which one is right depends on the grain of the covariates and on how far apart two records have to be before they are independent evidence.
Project, choose the distance from covariates, then thin
1. Project before thinning
A degree of longitude is 111 km at the equator and 50 km at 63°N. Thinning in degrees thins harder in the north, which is the opposite of what a northern species needs.
2. Choose the distance from the covariates, not the records
If the environmental layers are 1 km, records 200 m apart carry the same covariate values and one of them is redundant. That argument sets the scale; the record count does not.
3. Grid or distance — grid is usually enough
Grid thinning is a groupby and runs on millions of records. Distance-based thinning keeps a guaranteed minimum separation and costs a nearest-neighbour search per record.
4. Randomise which record survives
Keeping the first row keeps whichever record the API returned first, which correlates with dataset and date. A random choice within the cell is unbiased.
5. Repeat the model over several thinnings
Thinning is random. Fitting once on one thinned set hides how much the result depends on the draw.
6. Report the count before and after
Along with the distance, the CRS, and the seed.
Grid against distance thinning
The data used on this page
Occurrence records are pulled from the live GBIF occurrence search API by the code on this page. gbif.search is the paging function from How to download GBIF occurrences in Python, saved as a module by the setup. GBIF reprocesses continuously, so a later pull will differ in the last digits.
import os
os.makedirs("bio", exist_ok=True)
with open("gbif.py", "w") as f:
f.write('''
import pandas as pd, requests
API = "https://api.gbif.org/v1"
MAX_OFFSET = 100_000
FIELDS = ["key", "species", "speciesKey", "decimalLongitude", "decimalLatitude",
"coordinateUncertaintyInMeters", "year", "month", "basisOfRecord",
"datasetKey", "institutionCode", "issues"]
def count(**params):
return requests.get(f"{API}/occurrence/search", params={**params, "limit": 0}, timeout=120).json()["count"]
def search(limit=None, page=300, **params):
total = count(**params)
if limit is None and total > MAX_OFFSET:
raise ValueError(f"{total:,} records exceeds the {MAX_OFFSET:,} offset limit")
rows, offset = [], 0
target = total if limit is None else min(limit, total)
while offset < target:
body = requests.get(f"{API}/occurrence/search",
params={**params, "limit": min(page, target - offset), "offset": offset},
timeout=180).json()
rows += body["results"]
offset += len(body["results"])
if body["endOfRecords"] or not body["results"]:
break
return pd.DataFrame([{f: r.get(f) for f in FIELDS} for r in rows])
''')
print("gbif.py written")gbif.py written
Example 1 — grid thinning against distance thinning
from shapely import STRtree
def thin_by_distance(frame, min_m, seed=0):
"""Greedy: walk the records in random order, keep one if nothing kept is near."""
order = np.random.default_rng(seed).permutation(len(frame))
xs = frame.geometry.x.to_numpy()[order]
ys = frame.geometry.y.to_numpy()[order]
kept_x, kept_y, kept_idx = [], [], []
for i in range(len(xs)):
if kept_x:
d2 = (np.asarray(kept_x) - xs[i]) ** 2 + (np.asarray(kept_y) - ys[i]) ** 2
if d2.min() < min_m ** 2:
continue
kept_x.append(xs[i]); kept_y.append(ys[i]); kept_idx.append(order[i])
return frame.iloc[sorted(kept_idx)]
for d in (10_000, 25_000):
grid = thin_by_grid(pts, d)
dist = thin_by_distance(pts, d)
tree = STRtree(grid.geometry.values)
pairs = tree.query(grid.geometry.values, predicate="dwithin", distance=d)
touching = (pairs[0] != pairs[1]).sum() // 2
print(f"{d // 1000:>2} km: grid keeps {len(grid):>5,} "
f"({touching:>4} kept pairs closer than {d // 1000} km), "
f"distance keeps {len(dist):>5,} (0 by construction)")10 km: grid keeps 1,957 ( 536 kept pairs closer than 10 km), distance keeps 1,588 (0 by construction)
25 km: grid keeps 1,321 ( 616 kept pairs closer than 25 km), distance keeps 945 (0 by construction)
Grid thinning leaves pairs closer than the cell size — two records either side of a cell boundary are both kept. Distance thinning cannot, which is why it is the right choice when the minimum separation is the point rather than a proxy.
Example 2 — what thinning does to the clustering
def median_nn_km(frame):
"""Median distance from each record to its nearest other record."""
geoms = frame.geometry.values
tree = STRtree(geoms)
src, nbr = tree.query_nearest(geoms, exclusive=True, all_matches=False)
d = gpd.GeoSeries(geoms[src], crs=frame.crs).distance(
gpd.GeoSeries(geoms[nbr], crs=frame.crs), align=False)
return np.median(d) / 1000
print(f"{'dataset':<22}{'records':>9}{'median nearest neighbour':>28}")
print(f"{'raw':<22}{len(pts):>9,}{median_nn_km(pts):>25.2f} km")
for cell in (5_000, 10_000, 25_000):
kept = thin_by_grid(pts, cell)
print(f"{f'{cell // 1000} km grid':<22}{len(kept):>9,}{median_nn_km(kept):>25.2f} km")dataset records median nearest neighbour
raw 6,332 6.23 km
5 km grid 2,246 9.70 km
10 km grid 1,957 12.64 km
25 km grid 1,321 21.96 km
The raw median nearest-neighbour distance is 6.2 km, which sounds well spread until you remember that half the records are closer than that and some are metres apart. Each thinning lifts the median to roughly the cell size, which is what it is for: after a 10 km grid, the typical record's nearest neighbour is 12.6 km away.
Example 3 — how much the draw matters
counts, overlaps = [], []
base = set(thin_by_grid(pts, 10_000, seed=0).index)
for seed in range(1, 6):
kept = thin_by_grid(pts, 10_000, seed=seed)
counts.append(len(kept))
overlaps.append(len(base & set(kept.index)) / len(base))
print(f"10 km grid, six seeds: {min(counts):,}–{max(counts):,} records kept "
f"(identical counts: {len(set(counts)) == 1})")
print(f"share of the first draw's records also kept by later draws: "
f"{np.mean(overlaps):.0%} (min {min(overlaps):.0%})")
print("the count is fixed by the grid; which record survives each cell is not")10 km grid, six seeds: 1,957–1,957 records kept (identical counts: True)
share of the first draw's records also kept by later draws: 74% (min 74%)
the count is fixed by the grid; which record survives each cell is not
The number of records is a property of the grid, so it does not vary with the seed. Which record represents each cell does, and so do its covariate values — a record on a ridge and a record in the valley below can share a 10 km cell.
Why thinning helps, and why to repeat it
Why thinning helps at all
A model fitted on clustered presences sees the environment of the clusters repeatedly and treats it as strong evidence. Thinning equalises the weight of each area, so a well-recorded valley contributes once instead of four hundred times. It does nothing about places nobody visited.
Why the distance should come from the covariates
If the model's predictors are a 1 km climate grid, two records 300 m apart are the same observation as far as the model can tell. If they are a 30 m land-cover layer, they are not. The thinning distance is a statement about the resolution of the evidence, and choosing it by how many records survive is choosing the answer.
Why to repeat the thinning
Any single thinned dataset is one draw from a large set of equally valid datasets. Fitting the model on five or ten of them and reporting the spread of the results costs nothing but time, and it converts an arbitrary choice into a measured uncertainty.
Lost density, cell origins and latitude
- Thinning removes real density: for anything about abundance, thinning destroys the signal you want.
- Cell origin matters: a grid anchored at a different origin keeps a different set; that is part of the randomness.
- Latitude: use an equal-area or local projected CRS, not degrees.
- Temporal thinning — one record per cell per year — is the right form when the question is a time series.
- spThin (R) and
pyimpute-style helpers implement the distance variant with more options. - Keep the thinned-out records; the model needs the thinned set, and the evaluation may want all of them.
Internal links
- Sampling bias in occurrence data explained — the problem this addresses
- How to clean species occurrence records — the step before this one
- Species distribution models explained — what the thinned records feed
- A species distribution model scores well and predicts badly — why thinning is not enough
- Coordinate uncertainty explained — another reason nearby records are not independent
- How to aggregate points into H3 hexagons and map them — the same binning, for mapping
FAQ
What is spatial thinning?
Keeping at most one occurrence record per cell or per minimum distance, to reduce the effect of clustered sampling effort.
What distance should I use?
One set by the resolution of the covariates and by how far apart records must be to be independent — not by how many records survive.
Grid or distance based?
Grid is fast and leaves some close pairs; distance guarantees the separation and costs more. Both are defensible.
Does thinning remove sampling bias?
Partly. It removes the clustering; it cannot add records from places nobody visited.
Should I thin before or after cleaning?
After. Cleaning removes records that should never have been candidates for retention.
How do I report it?
Distance, CRS, seed, and the record count before and after.