⚙️ How-To Biodiversity & species data Data sources & downloads Biodiversity & Species Data · step 7 Notebook By · Updated

How to download GBIF occurrences in Python

On this page

Problem statement

The GBIF search API is easy to call and has one hard limit that decides how every real query has to be written: it refuses any offset above 100,000. A query matching more records than that cannot be paged to the end, and a loop that pages until it gets an error returns an arbitrary 100,000 records with no indication that it stopped early.

This guide pages within the limit, partitions a query that exceeds it, and shows where the asynchronous download API takes over.

Quick answer

Count first, then page — and fail loudly when the count is too big:

import time
import pandas as pd, requests

API = "https://api.gbif.org/v1"
MAX_OFFSET = 100_000
FIELDS = ["key", "species", "speciesKey", "decimalLongitude", "decimalLatitude",
          "coordinateUncertaintyInMeters", "year", "month", "basisOfRecord",
          "datasetKey", "institutionCode", "issues"]

def count(**params):
    return requests.get(f"{API}/occurrence/search",
                        params={**params, "limit": 0}, timeout=120).json()["count"]

def search(limit=None, page=300, **params):
    total = count(**params)
    if limit is None and total > MAX_OFFSET:
        raise ValueError(f"{total:,} records exceeds the {MAX_OFFSET:,} offset limit")
    rows, offset = [], 0
    target = total if limit is None else min(limit, total)
    while offset < target:
        body = requests.get(f"{API}/occurrence/search",
                            params={**params, "limit": min(page, target - offset),
                                    "offset": offset}, timeout=180).json()
        rows += body["results"]
        offset += len(body["results"])
        if body["endOfRecords"] or not body["results"]:
            break
    return pd.DataFrame([{f: r.get(f) for f in FIELDS} for r in rows])

bc = {"country": "CA", "stateProvince": "British Columbia", "hasCoordinate": "true"}
t0 = time.time()
bears = search(taxonKey=2433407, **bc)
print(f"{len(bears):,} records in {time.time() - t0:.1f}s")
print(f"all of British Columbia would be {count(**bc):,} records — "
      f"{count(**bc) / MAX_OFFSET:.0f}× the offset limit")
7,560 records in 21.4s
all of British Columbia would be 35,841,013 records — 358× the offset limit

A single species in one province is a comfortable query. The province itself is not, and no amount of paging will make it one.

Choosing a retrieval route from the record count. the count decides the routehow many records does the query match?under 100,000 — page thesearch APImore — partition by year ortaxonfor publication — thedownload APIOne species in one province: 7,560 records in 21 s. The province: 35.8 million. spatialworkflow.io
The count decides which of three routes the query takes.

Taxon key, count, page, partition past the limit

1. Resolve the name to a taxon key

Querying by scientificName matches strings; querying by taxonKey matches the backbone taxonomy, including synonyms and subspecies.

2. Count before you page

One request with limit=0 tells you whether the query is servable at all, and how long it will take.

3. Page with a sensible page size

300 records per request is a reasonable compromise. The API caps limit at 300 anyway.

4. Partition anything over the limit

By year, by taxon, by dataset, or by geographic tile. Each partition must be under 100,000 on its own, which means checking the counts, not assuming.

5. Use the download API for anything large or repeated

It is asynchronous, needs a free GBIF account, returns a DOI, and has no offset limit. Any analysis that will be published should use it.

6. Keep the query with the data

The parameters, the date, the record count and the dataset keys. Occurrence data changes daily as publishers add and correct records.

Three GBIF retrieval routes with their limits and what each returns. three routes, three limitsroutelimitwhat you getoccurrence/searchoffset <= 100,000records, immediately, no citationsearch with facetsno record transfer at allcounts and species lists per queryoccurrence/downloadnonea file and a DOI, asynchronouslylimit is capped at 300 per request whatever you passThe middle route is the one most scripts should be using and are not spatialworkflow.io
Three routes; the middle one is where most scripts silently go wrong.

The paging limit demonstrated, and partitioning by year

The data used on this page

Every number below comes from GBIF's API. GBIF adds and revises records daily, so the setup answers each GBIF request this page makes from the reply it got on 9 October 2026, and lets any other request through live.

import hashlib, io, zipfile

import requests

# GBIF adds and revises records daily. So that the numbers below reproduce, every GBIF request on
# this page is answered from the reply it got on 9 October 2026 (one file per request, named by a
# SHA-1 of its URL); any other request, or all of them once these lines are deleted, goes live.
SAVED = zipfile.ZipFile(io.BytesIO(requests.get(
    "https://data.spatialworkflow.io/gbif-api-2026-10-09.zip", timeout=300).content))
live_get = requests.get


def saved_get(url, params=None, **kwargs):
    prepared = requests.Request("GET", url, params=params).prepare()
    try:
        body = SAVED.read(hashlib.sha1(prepared.url.encode()).hexdigest()[:16] + ".json")
    except KeyError:
        return live_get(url, params=params, **kwargs)
    reply = requests.models.Response()
    reply.status_code, reply._content, reply.url = 200, body, prepared.url
    reply.headers["Content-Type"] = "application/json"
    return reply


requests.get = saved_get

Example 1 — the limit, demonstrated

for offset in (99_000, 100_000, 150_000):
    r = requests.get(f"{API}/occurrence/search",
                     params={**bc, "limit": 1, "offset": offset}, timeout=120)
    print(f"offset {offset:>7,}: HTTP {r.status_code}"
          + ("" if r.status_code == 200 else f" — {r.text[:60]}"))
offset  99,000: HTTP 200
offset 100,000: HTTP 200
offset 150,000: HTTP 400 — Max offset of 100001 exceeded: 150000 + 1

The failure is at least an HTTP error rather than a truncated page — but only if you look at the status code, which a requests.get(...).json() in a loop does not.

Example 2 — partitioning by year

def partition_by_year(years, **params):
    parts, sizes = [], {}
    for year in years:
        n = count(**params, year=year)
        sizes[year] = n
        if n == 0:
            continue
        if n > MAX_OFFSET:
            raise ValueError(f"{year} alone has {n:,} records — partition further")
        parts.append(search(**params, year=year))
    return pd.concat(parts, ignore_index=True) if parts else pd.DataFrame(), sizes

mammals = {"country": "CA", "stateProvince": "British Columbia",
           "taxonKey": 359, "hasCoordinate": "true"}          # class Mammalia
total = count(**mammals)
print(f"all BC mammal records: {total:,}")
per_year = {y: count(**mammals, year=y) for y in range(2019, 2026)}
print("per year: " + ", ".join(f"{y}: {n:,}" for y, n in per_year.items()))
print(f"largest single year: {max(per_year.values()):,} "
      f"({'servable' if max(per_year.values()) <= MAX_OFFSET else 'still too big'})")
all BC mammal records: 133,305
per year: 2019: 5,044, 2020: 8,190, 2021: 9,892, 2022: 11,590, 2023: 13,521, 2024: 16,047, 2025: 19,169
largest single year: 19,169 (servable)

A hundred and thirty thousand mammal records is 1.3 times the limit, and splitting by year fixes it with room to spare — the busiest single year is 19,169. The skew is the useful part: records per year have nearly quadrupled since 2019, so the partition that will break first is next year's, not an old one.

Example 3 — the download API, for anything that matters

request_body = {
    "creator": "your-gbif-username",
    "notificationAddresses": ["you@example.org"],
    "format": "SIMPLE_CSV",
    "predicate": {"type": "and", "predicates": [
        {"type": "equals", "key": "COUNTRY", "value": "CA"},
        {"type": "equals", "key": "STATE_PROVINCE", "value": "British Columbia"},
        {"type": "equals", "key": "HAS_COORDINATE", "value": "true"},
        {"type": "equals", "key": "TAXON_KEY", "value": "359"},
    ]},
}
print("POST /occurrence/download/request with HTTP basic auth")
print("  -> returns a download key; poll /occurrence/download/{key} for status")
print("  -> the finished download has a DOI, which is what a paper cites")
print(f"\nthe same query through the search API would need "
      f"{count(**mammals) // 300:,} requests and still hit the offset limit")
print("predicate keys are upper snake case and differ from the search parameters:")
for search_param, predicate_key in [("country", "COUNTRY"), ("taxonKey", "TAXON_KEY"),
                                    ("hasCoordinate", "HAS_COORDINATE"),
                                    ("stateProvince", "STATE_PROVINCE")]:
    print(f"  {search_param:16} -> {predicate_key}")
POST /occurrence/download/request with HTTP basic auth
  -> returns a download key; poll /occurrence/download/{key} for status
  -> the finished download has a DOI, which is what a paper cites

the same query through the search API would need 444 requests and still hit the offset limit
predicate keys are upper snake case and differ from the search parameters:
  country          -> COUNTRY
  taxonKey         -> TAXON_KEY
  hasCoordinate    -> HAS_COORDINATE
  stateProvince    -> STATE_PROVINCE

The download route needs a free account, so this example prints the request rather than sending it. Everything above it runs anonymously, which is what makes the search API the right tool for exploration and the wrong one for a result.

Why the offset limit exists

Why the offset limit exists

Deep paging over a distributed index means every shard walks to the offset on every request, so a query at offset two million costs the server as much as returning two million records. Capping the offset is how search engines stay fast, and the asynchronous download path exists precisely because the deep case is expensive enough to be worth queueing.

Why partitioning is not just splitting

Any partition works as long as the parts are disjoint, cover the whole query, and each stays under the limit. Year is convenient because occurrence data is heavily skewed towards recent years — which also means that the partition that breaks the limit first is usually the current one, not the oldest.

Why the data is a moving target

Publishers add records, correct coordinates and retract datasets continuously, and GBIF re-runs its interpretation over everything. The same query a month apart returns different counts. A download DOI is a snapshot; a search API query is not, which is why the parameters and the date belong beside the results.

One query over the offset limit split into per-year partitions. one impossible query, seven servable ones133,305 records — over the limit20195,04420208,19020219,892202211,590202313,521202416,047202519,169each part is checked against its own count, and the parts against the wholethe busiest year is 19,169 — a fifth of the limit, with room to grow spatialworkflow.io
Partitioning turns one impossible query into several servable ones.

Page caps, rate limits and child taxa

  • limit is capped at 300 per request; larger values are silently reduced.
  • Rate limiting is polite rather than strict; add a pause in a long loop and set a real User-Agent.
  • taxonKey includes children, so a genus key returns all its species.
  • geometry takes a WKT polygon in WGS 84 with counter-clockwise winding — the winding trips people up.
  • occurrenceStatus=PRESENT excludes the absence records some datasets publish.
  • pygbif wraps all of this and has the same limits underneath.

FAQ

How do I download GBIF records in Python?

Count first, then page the search API with limit and offset, staying under the 100,000 offset limit.

What is GBIF's record limit?

The search API refuses any offset above 100,000 — the error says "Max offset of 100001 exceeded". The asynchronous download API has no such limit.

How do I get more than 100,000 records?

Partition the query — by year, taxon or area — or use the download API, which needs a free account and returns a DOI.

What page size should I use?

300, which is the maximum the API accepts.

Should I query by name or by taxon key?

By taxon key: it resolves synonyms and includes child taxa.

Will the same query give the same answer tomorrow?

No. Publishers change records continuously; cite a download DOI if the result matters.