How to clean species occurrence records

On this page

Problem statement

Occurrence data arrives with duplicates, coordinates at zero, positions at country centroids, specimens georeferenced from a place name with a ten-kilometre radius, and records from 1860 sitting next to records from last week. None of it is wrong as a record — a museum catalogue entry with a vague locality is a true statement about a specimen — but almost all of it is wrong as an input to a distribution model.

Cleaning is a sequence of explicit filters, each of which removes a named class of problem, each of which is a decision. On 6,353 georeferenced Canada lynx records, a standard sequence removes three quarters of them.

Quick answer

Filter in named steps and count what each one costs:

import numpy as np, pandas as pd
from gbif import search

lynx = search(taxonKey=2435263, hasCoordinate="true")        # Lynx canadensis, worldwide
print(f"{len(lynx):,} georeferenced records")

steps = {}
keep = lynx.copy()
steps["start"] = len(keep)

keep = keep[~((keep.decimalLatitude.abs() < 0.01) & (keep.decimalLongitude.abs() < 0.01))]
steps["drop 0,0 coordinates"] = len(keep)

bad = {"ZERO_COORDINATE", "COORDINATE_OUT_OF_RANGE", "COUNTRY_COORDINATE_MISMATCH",
       "COORDINATE_INVALID", "GEODETIC_DATUM_INVALID"}
keep = keep[~keep.issues.map(lambda r: bool(bad & set(r or [])))]
steps["drop records with fatal flags"] = len(keep)

keep = keep[keep.coordinateUncertaintyInMeters.fillna(0) <= 10_000]
steps["uncertainty <= 10 km or missing"] = len(keep)

keep = keep[keep.year >= 1970]
steps["1970 onwards"] = len(keep)

keep = keep.drop_duplicates(subset=["species", "decimalLatitude", "decimalLongitude", "year"])
steps["drop duplicate position-years"] = len(keep)

prev = None
for name, n in steps.items():
    delta = "" if prev is None else f"  (−{prev - n:,})"
    print(f"{name:34} {n:>6,}{delta}")
    prev = n
print(f"\nsurvivors: {len(keep) / len(lynx):.1%} of the input")

6,353 georeferenced records
start                               6,353
drop 0,0 coordinates                6,350  (−3)
drop records with fatal flags       6,007  (−343)
uncertainty <= 10 km or missing     4,077  (−1,930)
1970 onwards                        2,174  (−1,903)
drop duplicate position-years       1,627  (−547)

survivors: 25.6% of the input

The two expensive filters are not the ones that sound expensive. Fatal flags cost 343 records; uncertainty and date cost 1,930 and 1,903 — nearly two thirds of the input between them, because this species has a century of museum specimens georeferenced from place names. Every one of those lines is arguable. Publishing the table is what makes them reviewable.

Map of North America with 6,353 Canada lynx records coloured by the cleaning step that removes them: kept records in teal across Alaska, western Canada and the Great Lakes; pre-1970 records in orange mostly in the west and north; imprecise records in blue in clusters; flagged records in red near the Great Lakes. — Every record, coloured by the step that removes it. The survivors (teal) still cover the range; what goes is not random. The pre-1970 records are mostly in the west and north; the imprecise ones come in tight clusters, which is what a batch of records given one coarse coordinate looks like; the flagged ones concentrate in a few places near the Great Lakes, a sign of one dataset with one problem.

Six filters on one species: 6,353 records in, 1,627 out.

Aggregator flags, missing uncertainty, a defensible duplicate key

1. Start from the flags the aggregator already computed

GBIF's issues list catches zero coordinates, out-of-range values, country mismatches and invalid datums. Reading them is cheaper than re-deriving them.

2. Decide what to do with missing uncertainty

A third of these records have no coordinateUncertaintyInMeters. Dropping them removes most of the museum material; keeping them means accepting unknown precision. Choose, and say which.

3. Remove duplicates on a defensible key

Species, position and year catches the same specimen catalogued twice. A stricter key keeps genuine repeat observations; a looser one throws them away.

4. Cut by date if the question has a date

A model of current distribution should not be fitted on records from before the species was extirpated from half its range.

5. Check the positions against land

A terrestrial species in the sea is a georeferencing error, a transposed sign, or a coastal record that a coarse land mask cannot resolve.

6. Keep the removed records

Write them out with the reason. Cleaning that cannot be inspected is indistinguishable from cleaning that is wrong.

Eight checks; each one is a judgement about what the data is for.

Cleaning Canada lynx records

The data used on this page

Canada lynx (Lynx canadensis, GBIF taxon key 2435263) occurrence records with coordinates, worldwide, pulled from the GBIF occurrence search API. gbif.search is the paging function from How to download GBIF occurrences in Python, saved as a module by the setup. GBIF reprocesses continuously, so the setup answers each request with the reply it got on 9 October 2026; a live pull differs in the last digits.

import hashlib
import io
import os
import zipfile

import requests

# GBIF adds and revises records daily. So that the numbers below reproduce, every GBIF request on
# this page is answered from the reply it got on 9 October 2026 (one file per request, named by a
# SHA-1 of its URL); any other request, or all of them once these lines are deleted, goes live.
SAVED = zipfile.ZipFile(io.BytesIO(requests.get(
    "https://data.spatialworkflow.io/gbif-api-2026-10-09.zip", timeout=300).content))
live_get = requests.get


def saved_get(url, params=None, **kwargs):
    prepared = requests.Request("GET", url, params=params).prepare()
    try:
        body = SAVED.read(hashlib.sha1(prepared.url.encode()).hexdigest()[:16] + ".json")
    except KeyError:
        return live_get(url, params=params, **kwargs)
    reply = requests.models.Response()
    reply.status_code, reply._content, reply.url = 200, body, prepared.url
    reply.headers["Content-Type"] = "application/json"
    return reply


requests.get = saved_get

os.makedirs("bio", exist_ok=True)
with open("gbif.py", "w") as f:
    f.write('''
import pandas as pd, requests

API = "https://api.gbif.org/v1"
MAX_OFFSET = 100_000
FIELDS = ["key", "species", "speciesKey", "decimalLongitude", "decimalLatitude",
          "coordinateUncertaintyInMeters", "year", "month", "basisOfRecord",
          "datasetKey", "institutionCode", "issues"]


def count(**params):
    return requests.get(f"{API}/occurrence/search", params={**params, "limit": 0}, timeout=120).json()["count"]


def search(limit=None, page=300, **params):
    total = count(**params)
    if limit is None and total > MAX_OFFSET:
        raise ValueError(f"{total:,} records exceeds the {MAX_OFFSET:,} offset limit")
    rows, offset = [], 0
    target = total if limit is None else min(limit, total)
    while offset < target:
        body = requests.get(f"{API}/occurrence/search",
                            params={**params, "limit": min(page, target - offset), "offset": offset},
                            timeout=180).json()
        rows += body["results"]
        offset += len(body["results"])
        if body["endOfRecords"] or not body["results"]:
            break
    return pd.DataFrame([{f: r.get(f) for f in FIELDS} for r in rows])
''')
print("gbif.py written")

gbif.py written

Example 1 — what the flags say before you filter

flags = pd.Series([i for row in lynx.issues.dropna() for i in row]).value_counts()
print(flags.head(8).to_string())

fatal = {"ZERO_COORDINATE", "COORDINATE_OUT_OF_RANGE", "COUNTRY_COORDINATE_MISMATCH",
         "COORDINATE_INVALID", "GEODETIC_DATUM_INVALID"}
hit = lynx.issues.map(lambda r: sorted(fatal & set(r or [])))
print(f"\nrecords carrying at least one fatal flag: {(hit.map(len) > 0).sum():,}")
print(pd.Series([f for row in hit for f in row]).value_counts().to_string())

CONTINENT_DERIVED_FROM_COORDINATES                  5685
AMBIGUOUS_INSTITUTION                               3281
COORDINATE_ROUNDED                                  2926
TAXON_ID_NOT_FOUND                                  1776
GEODETIC_DATUM_ASSUMED_WGS84                         841
INSTITUTION_MATCH_FUZZY                              716
OCCURRENCE_STATUS_INFERRED_FROM_INDIVIDUAL_COUNT     692
INSTITUTION_COLLECTION_MISMATCH                      478

records carrying at least one fatal flag: 346
GEODETIC_DATUM_INVALID         342
ZERO_COORDINATE                  3
COUNTRY_COORDINATE_MISMATCH      2

Example 2 — duplicates, and how many there really are

keys = {
    "position + year": ["decimalLatitude", "decimalLongitude", "year"],
    "position + date": ["decimalLatitude", "decimalLongitude", "year", "month"],
    "position only": ["decimalLatitude", "decimalLongitude"],
    "position + dataset": ["decimalLatitude", "decimalLongitude", "datasetKey"],
}
for name, cols in keys.items():
    dup = lynx.duplicated(subset=cols).sum()
    print(f"{name:22} {dup:>5,} duplicates ({dup / len(lynx):.0%})")

worst = (lynx.groupby(["decimalLatitude", "decimalLongitude"]).size()
         .sort_values(ascending=False).head(3))
print("\nmost repeated positions:")
for (lat, lon), n in worst.items():
    sub = lynx[(lynx.decimalLatitude == lat) & (lynx.decimalLongitude == lon)]
    print(f"  {lat:8.4f}, {lon:9.4f}: {n:>4} records, "
          f"{sub.year.min():.0f}–{sub.year.max():.0f}, "
          f"{sub.datasetKey.nunique()} dataset(s)")

position + year        3,045 duplicates (48%)
position + date        2,632 duplicates (41%)
position only          3,597 duplicates (57%)
position + dataset     3,588 duplicates (56%)

most repeated positions:
   61.3500, -117.6500:  261 records, 1998–1999, 1 dataset(s)
   61.6558, -145.1753:  119 records, 1963–1972, 1 dataset(s)
   66.5647, -145.2739:  115 records, 1963–2000, 1 dataset(s)

A position with hundreds of records and a single dataset is usually a georeferenced place name — a park, a township, a museum — not a place where hundreds of lynx were seen.

Example 3 — records that are not on land

import geopandas as gpd, io, os, requests, zipfile

if not os.path.exists("bio/ne_land.gpkg"):
    blob = requests.get("https://naciscdn.org/naturalearth/110m/physical/ne_110m_land.zip",
                        timeout=120).content
    zipfile.ZipFile(io.BytesIO(blob)).extractall("bio/ne_land")
    gpd.read_file("bio/ne_land/ne_110m_land.shp").to_file("bio/ne_land.gpkg", driver="GPKG")
land = gpd.read_file("bio/ne_land.gpkg")

pts = gpd.GeoDataFrame(lynx, crs=4326,
                       geometry=gpd.points_from_xy(lynx.decimalLongitude, lynx.decimalLatitude))
on_land = gpd.sjoin(pts, land, predicate="within", how="left").index_right.notna().values
sea = pts[~on_land]
print(f"{len(sea):,} of {len(pts):,} records fall outside the 110 m land polygons")

coast = land.to_crs(3857).boundary.union_all()
dist = sea.to_crs(3857).distance(coast)
print(f"  within 10 km of the coastline: {(dist < 10_000).sum():,} — a coarse-mask artefact")
print(f"  more than 50 km offshore:      {(dist > 50_000).sum():,} — a real error")
if (dist > 50_000).any():
    far = sea[dist.values > 50_000].head(3)
    print(far[["decimalLatitude", "decimalLongitude", "year",
               "basisOfRecord"]].to_string(index=False))

68 of 6,353 records fall outside the 110 m land polygons
  within 10 km of the coastline: 33 — a coarse-mask artefact
  more than 50 km offshore:      7 — a real error
 decimalLatitude  decimalLongitude   year      basisOfRecord
       53.105452        -80.957822 2025.0  HUMAN_OBSERVATION
       52.121667        177.558611 2014.0 PRESERVED_SPECIMEN
       53.125053        -80.820097 2008.0  HUMAN_OBSERVATION

The coarse land mask puts every island and fjord record in the sea, so the distance to the coastline is what separates a mask artefact from a genuinely impossible position.

Cleaning is subtraction, and subtraction is a model

Why cleaning is subtraction, and subtraction is a model

Each filter encodes an assumption about what the analysis needs: that positions are precise to a kilometre, that the last fifty years represent the present, that the same coordinates twice in one year is one event. Those assumptions belong in the write-up, because a different question would justify different ones — a historical range reconstruction wants exactly the records a distribution model throws away.

Why duplicates are the biggest single cut

Specimens are catalogued by the holding institution, by a national aggregator and by a digitisation project, and each publishes to GBIF. Observations are uploaded to a platform and re-published by a regional portal. The result is that half of a typical pull is the same evidence more than once, which biases everything downstream towards whichever records were duplicated most.

Why keeping the rejects matters

A rejected record is evidence about the cleaning, not just about the species. Ten records rejected for being in the sea, all from one dataset, all with transposed signs, is a bug worth reporting to the publisher. The same ten silently dropped is a slightly better map and a slightly worse dataset for everyone else.

How many duplicates there are depends entirely on what you call one.

Country centroids, sign errors and re-downloads

coordinatecleaner (R) and its Python equivalents test for biodiversity institution coordinates, country centroids and capitals; worth running as well as the flags.
Country centroids are a specific, common failure: a whole country's records at one point.
Sign transposition puts records in the ocean or the wrong hemisphere; check for mirrored clusters.
Rounded coordinates are not errors, but they cluster records onto a grid, which matters for thinning.
Sensitive species have deliberately generalised coordinates; removing them by uncertainty removes the species.
Re-run cleaning when the data is re-downloaded; the flags change as GBIF reprocesses.

Internal links

Species occurrence data explained — the fields these filters read
How to download GBIF occurrences in Python — getting the records
Coordinate uncertainty explained — the filter that costs the most records
Occurrence records fall in the ocean — the fix guide
How to thin occurrence records spatially — the cleaning step models need
How to clean a line network: dangles, overshoots and disconnects — the same argument for geometry

FAQ

How do I clean GBIF records?

Filter in named steps — flags, uncertainty, date, duplicates, land — and report how many records each step removed.

Which GBIF issue flags should I treat as fatal?

Zero coordinates, out-of-range values, invalid datum and country mismatches. The rest are warnings.

How do I find duplicates?

Pick a key — species, position and year is a reasonable default — and say which one you used, because the count depends on it.

Should I drop records with no uncertainty?

It depends on the question. Dropping them removes most museum material; keeping them means unknown precision.

Why are records in the sea?

Coarse land masks, coastal positions, transposed signs, or genuinely wrong coordinates. Distance offshore separates them.

Do I need to keep the rejected records?

Yes — they are the only evidence that the cleaning did what you think it did.