🔧 Fix Biodiversity & species data Data sources & downloads Biodiversity & Species Data · step 16 Notebook By · Updated

A GBIF download is missing records

On this page

Problem statement

The script pulled 100,000 records and the analysis ran. GBIF's website says the query matches 380,000. Nothing errored, nothing warned, and the difference will never appear in the output — it will appear as a species that seems to be absent from half its range.

There are four ways to lose records from a GBIF query, and only one of them produces an error message. The others are a silent offset cap, a filter that means something other than what it looks like, and a dataset that changed between the two times you ran the query.

Quick answer

Compare what you got with what the API says exists, every time:

import requests

API = "https://api.gbif.org/v1/occurrence/search"

def expected(**params):
    return requests.get(API, params={**params, "limit": 0}, timeout=120).json()["count"]

def fetch(page=300, **params):
    got, offset = [], 0
    while True:
        r = requests.get(API, params={**params, "limit": page, "offset": offset},
                         timeout=180)
        if r.status_code != 200:                       # the loop many scripts omit
            raise RuntimeError(f"HTTP {r.status_code} at offset {offset:,}: "
                               f"{r.text[:80]}")
        body = r.json()
        got += body["results"]
        offset += len(body["results"])
        if body["endOfRecords"] or not body["results"]:
            break
    return got

query = {"country": "CA", "stateProvince": "Ontario",
         "taxonKey": 2433407, "hasCoordinate": "true"}
want = expected(**query)
print(f"the API says {want:,} records match")
got = fetch(**query)
print(f"retrieved {len(got):,}")
print(f"complete: {len(got) == want}")
the API says 3,639 records match
retrieved 3,639
complete: True

The assertion is the point. A pull that cannot state how many records it should have cannot know that it is short.

Four ways a GBIF pull loses records, with the symptom and the fix. why the pull is shortWhat happenedHow to catch itthe offset cap stopped the loop at100,000Compare the retrieved count withthe API’s own counta non-200 response was swallowedby a try/exceptRaise on r.status_code != 200inside the loopa filter removed more thanexpectedCount after each filter:hasCoordinate costs 45% herethe index changed since the lastrunRecord the date and the count;cite a download DOI spatialworkflow.io
Four losses; only the first announces itself.

Count before, check after, partition past the cap

1. Count before and check after

One extra request. Without it, "the loop finished" is the only evidence the data is complete.

2. Raise on a non-200 response

At offsets above 100,000 the API returns HTTP 400 with a body that is not the usual object — sometimes a JSON list of error strings, sometimes plain text. A script that calls .json()["results"] on it raises; one that catches the exception and breaks stops early and says nothing.

3. Partition rather than page past the cap

By year, taxon, dataset or tile, with each part checked against its own count.

4. Know what the filters exclude

hasCoordinate=true drops records with no coordinates — often a third of a species' records, and disproportionately the historical ones. hasGeospatialIssue=false drops more.

5. Use the download API when the result matters

It has no offset limit, returns a DOI, and produces the same answer when re-run.

6. Record the date and the counts

The index changes daily. A query rerun next month legitimately returns a different number, and without the earlier count you cannot tell that from a bug.

Record counts as each common GBIF filter is applied. what the filters cost, Canada lynxall records11,568hasCoordinatewith coordinates6,353 (−45%)flagsno geospatial issues6,348year÷ 2.4since 19902,683 (−58%)basishuman observations1,498 (−44%)Four filters, each defensible, leave one record in eight spatialworkflow.io
Each filter is a decision; together they remove more than most people expect.

What each filter and the cap cost

The data used on this page

Every number below comes from GBIF's API. GBIF adds and revises records daily, so the setup answers each GBIF request this page makes from the reply it got on 9 October 2026, and lets any other request through live.

import hashlib, io, zipfile

import requests

# GBIF adds and revises records daily. So that the numbers below reproduce, every GBIF request on
# this page is answered from the reply it got on 9 October 2026 (one file per request, named by a
# SHA-1 of its URL); any other request, or all of them once these lines are deleted, goes live.
SAVED = zipfile.ZipFile(io.BytesIO(requests.get(
    "https://data.spatialworkflow.io/gbif-api-2026-10-09.zip", timeout=300).content))
live_get = requests.get


def saved_get(url, params=None, **kwargs):
    prepared = requests.Request("GET", url, params=params).prepare()
    try:
        body = SAVED.read(hashlib.sha1(prepared.url.encode()).hexdigest()[:16] + ".json")
    except KeyError:
        return live_get(url, params=params, **kwargs)
    reply = requests.models.Response()
    reply.status_code, reply._content, reply.url = 200, body, prepared.url
    reply.headers["Content-Type"] = "application/json"
    return reply


requests.get = saved_get

Example 1 — what each filter actually costs

base = {"taxonKey": 2435263}                             # Lynx canadensis, worldwide
steps = [
    ("all records", {}),
    ("+ hasCoordinate", {"hasCoordinate": "true"}),
    ("+ no geospatial issues", {"hasCoordinate": "true", "hasGeospatialIssue": "false"}),
    ("+ since 1990", {"hasCoordinate": "true", "hasGeospatialIssue": "false",
                      "year": "1990,2026"}),
    ("+ human observations", {"hasCoordinate": "true", "hasGeospatialIssue": "false",
                              "year": "1990,2026",
                              "basisOfRecord": "HUMAN_OBSERVATION"}),
]
prev = None
for label, extra in steps:
    n = expected(**base, **extra)
    delta = "" if prev is None else f"  (−{prev - n:,}, −{1 - n / prev:.0%})"
    print(f"{label:<24}{n:>8,}{delta}")
    prev = n
all records               11,568
+ hasCoordinate            6,353  (−5,215, −45%)
+ no geospatial issues     6,348  (−5, −0%)
+ since 1990               2,683  (−3,665, −58%)
+ human observations       1,498  (−1,185, −44%)

Example 2 — the cap, and what a naive loop does with it

big = {"country": "CA", "stateProvince": "Ontario", "taxonKey": 359,
       "hasCoordinate": "true"}
print(f"matching records: {expected(**big):,}")

for offset in (99_000, 150_000):
    r = requests.get(API, params={**big, "limit": 1, "offset": offset}, timeout=120)
    print(f"offset {offset:>7,}: HTTP {r.status_code}, "
          f"content-type {r.headers.get('content-type', '?').split(';')[0]}")
    try:
        body = r.json()
        kind = type(body).__name__
        print(f"    body parses as {kind}; results present: "
              f"{isinstance(body, dict) and 'results' in body}")
    except Exception as e:
        print(f"    body is not JSON ({type(e).__name__}); text: {r.text[:70]!r}")
matching records: 210,913
offset  99,000: HTTP 200, content-type application/json
    body parses as dict; results present: True
offset 150,000: HTTP 400, content-type text/plain
    body is not JSON (JSONDecodeError); text: 'Max offset of 100001 exceeded: 150000 + 1'

Ontario's mammals, 210,913 records, are more than twice the cap, so no page size reaches the end of them. The refusal is explicit in the status code and nowhere else that a parser will notice: the body is not even JSON. A loop wrapped in try/except Exception: break turns that into a silent truncation at whatever offset the cap happens to be, which is why the status check belongs in the loop rather than around it.

Example 3 — partition, and verify each part

years = range(2015, 2026)
parts, total = {}, 0
for y in years:
    n = expected(**big, year=y)
    parts[y] = n
    total += n
    if n > 100_000:
        print(f"  {y}: {n:,} — still over the cap, partition further")
print(f"sum of yearly counts: {total:,}")
print(f"whole-query count:    {expected(**big):,}")
print(f"records outside {min(years)}–{max(years)}: "
      f"{expected(**big) - total:,}")
print("  " + ", ".join(f"{y}: {n:,}" for y, n in parts.items()))
sum of yearly counts: 148,551
whole-query count:    210,913
records outside 2015–2025: 62,362
  2015: 1,461, 2016: 2,208, 2017: 3,520, 2018: 5,633, 2019: 8,304, 2020: 14,135, 2021: 19,341, 2022: 18,209, 2023: 19,804, 2024: 25,395, 2025: 30,541

The yearly counts sum to less than the whole, and the difference is the records from before 2015 — which is the check that the partition covers the query. A partition that does not reconcile is a partition with a gap in it.

Why the cap is silent

Why the cap exists and why it is silent

Deep paging is expensive in a distributed index, so the API refuses offsets above 100,000 with an HTTP 400. The refusal is explicit; what makes it silent is the body, which is not the usual object and is sometimes not JSON at all, so the failure surfaces as a parse error inside the caller rather than as "your query was too big".

Why hasCoordinate costs more than it looks

Records without coordinates are not a random sample: they are older, more likely to be museum specimens, and more likely to be from regions with less digitisation. Filtering them out is usually correct for a spatial analysis and quietly shifts the data towards recent citizen-science records in well-digitised countries.

Why re-running gives a different number

Publishers add datasets, correct coordinates and retract records continuously, and GBIF reinterprets everything against a changing backbone taxonomy. A count from last month is not reproducible by re-querying; it is reproducible by citing a download DOI, which is what the download API exists to give you.

A query over the offset limit partitioned by year with each part checked. the partition has to reconcile with the wholewhole query: 210,9132015–2025: 148,551difference: 62,362 before 2015a partition that does not reconcile has a gap in itand a gap in a partition is a silent missing slice of the data spatialworkflow.io
Each part is checked against its own count, and the parts are checked against the whole.

Limits, geometry filters and website counts

  • limit above 300 is silently reduced; a loop that assumes 1,000 per page will make three times as many requests as it expects.
  • geometry needs counter-clockwise WKT; a clockwise polygon selects the rest of the world and returns far too many records.
  • Rate limiting returns HTTP 429 with a list body, the same shape trap as the offset error.
  • pygbif wraps the API and inherits its limits; occ.download is the one that does not.
  • DOIs are issued per download and are what a paper cites; a search API query has nothing to cite.
  • Counts differ between the website and the API when the website applies a default filter you did not.

FAQ

Why did my GBIF pull stop at 100,000 records?

The search API refuses offsets above 100,000. Partition the query or use the download API.

Why did it not raise an error?

It does — as an HTTP 400 whose body is a list, which many scripts turn into a silent break.

How do I know a pull is complete?

Compare the number of records retrieved with the count the API reports for the same filters.

Does hasCoordinate=true lose much?

Often a third of the records, and disproportionately the older ones.

Why does the count change between runs?

Publishers and GBIF update records continuously. Cite a download DOI if the number matters.

Can I page past the limit with a smaller page size?

No — the limit is on the offset, not on the page.