🧭 Concept Data sources & downloads Getting started Finding & Fetching Data · step 1 of 14 Updated

GIS Data Sources Explained: Where Spatial Data Comes From

Problem statement

Every tutorial hands you a file. Real work starts a step earlier, with a question nobody has written the answer to: where does this layer come from?

That question is not idle curiosity. It decides four things you will be held to later:

  • Whether you are allowed to publish the result. Some sources require attribution. Some require that derived data be shared under the same licence. Some forbid commercial use entirely.
  • Whether the answer is current. A boundary file downloaded in 2019 will happily produce a 2026 report, and nothing in the code will notice.
  • Whether the coverage matches the question. A national dataset that stops at a coastline is fine until your study area includes an estuary.
  • Whether the granularity supports the claim. Building footprints answer "how many houses"; a 1 km population grid does not, no matter how you slice it.

The failure mode is not an error. It is a finished map that is wrong in a way only the data provider could have told you about.

Quick answer

Spatial data comes from four families, and they fail differently:

Family Examples Strength The catch
Authoritative national mapping agency, census, cadastre defined accuracy, legal standing slow to update, portal downloads, restrictive licences
Crowdsourced OpenStreetMap current, global, one schema coverage varies by place, completeness unknown
Derived / commercial address matching, demographics, road networks cleaned and joined for you opaque method, licence-locked, costs money
Sensor / observation satellite imagery, LiDAR, GPS traces measures the world, not a model of it huge, needs processing, gaps from clouds and flight lines

Before you use any of them, record four facts alongside the file:

PROVENANCE = {
    "source": "OpenStreetMap via Overpass API",
    "licence": "ODbL 1.0",
    "retrieved": "2026-08-26",
    "extent": "bbox 53.4770,-2.2500,53.4850,-2.2350",
}

Four lines. They are the difference between a dataset and a folder of mystery files.

Four families of spatial data source — authoritative, crowdsourced, derived and sensor — compared on currency, coverage and licence freedom.
No family wins on all three. Choosing a source is choosing which weakness you can live with.

Step-by-step solution

1. Start from the claim, not the catalogue

Write down the sentence the map is meant to support before you go looking. "Forty percent of residents live more than 800 m from a park" fixes three requirements immediately: you need residents (a population layer with a defined denominator), parks (a boundary layer with a definition of park), and distance (network or straight-line — they differ).

Browsing a data portal first produces the opposite: a dataset you have, and a claim bent to fit it.

2. Identify which family can answer it

Run the claim past the four families:

  • Does it need legal or official standing? Administrative boundaries, land ownership and statistical geographies must be authoritative. OpenStreetMap boundaries are usually correct but carry no authority — do not use them to say where a council's responsibility ends.
  • Does it need to be current? Crowdsourced data is often months ahead of the official release. A new roundabout appears in OSM the week it opens.
  • Does it need measurement rather than record? Anything about vegetation, water, heat or built-up extent is a sensor question.
  • Does it need coverage you cannot assemble? That is where commercial data earns its price.

3. Check the licence before you check the data

Read the licence page, not the download button. Three phrases decide most of it:

  • Attribution required — you must name the source on the output. Fine.
  • Share-alike (ODbL, CC BY-SA) — derived data inherits the licence. Fine for a public map, fatal for a client deliverable that must be proprietary.
  • Non-commercial — excludes most consultancy work, including work that is merely paid for.

"Open data" is not one thing. OpenStreetMap is ODbL: share-alike. Most national statistics are attribution-only. Some "open" portals are open to view and closed to redistribute.

4. Choose the access mode

The same dataset is often offered several ways, and the choice sets how much of your pipeline is code:

Four ways to get the same layer — bulk file, API query, live web service, and rendered tiles — ordered by how much data you receive and how much control you keep.
Further down the ladder you get less data and less control, but far less to maintain.
  • Bulk file (a national GeoPackage, a shapefile ZIP). Everything at once. Best when you need the whole thing repeatedly; worst when it is 8 GB and you need one city.
  • API query (Overpass, an ArcGIS REST endpoint). You describe what you want and get only that. Best for a defined study area. Introduces pagination, rate limits and downtime.
  • Live service (WFS, OGC API Features). Same as an API, but standardised — so one client works against many providers.
  • Rendered tiles (a basemap). Pixels, not features. You cannot analyse them; you can only put them behind your data.

If the answer needs numbers, you need one of the first three. A basemap is context, never evidence.

5. Record provenance at fetch time

The only moment you reliably know where a file came from is the moment you download it. Write it down then, in a machine-readable sidecar, not in a README you will forget.

Code examples

Example 1 — a fetch wrapper that cannot forget provenance 

import hashlib
import json
from datetime import date, timezone, datetime
from pathlib import Path

import requests


def fetch(url, dest, *, source, licence, params=None, headers=None):
    """Download `url` to `dest` and write `dest.json` describing where it came from."""
    dest = Path(dest)
    dest.parent.mkdir(parents=True, exist_ok=True)

    response = requests.get(url, params=params, headers=headers, timeout=120)
    response.raise_for_status()
    dest.write_bytes(response.content)

    meta = {
        "source": source,
        "licence": licence,
        "url": response.url,           # after redirects and parameter encoding
        "retrieved": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "bytes": len(response.content),
        "sha256": hashlib.sha256(response.content).hexdigest(),
        "status": response.status_code,
    }
    dest.with_suffix(dest.suffix + ".json").write_text(json.dumps(meta, indent=2))
    return dest


path = fetch(
    "https://demo.pygeoapi.io/master/collections/lakes/items",
    "data/raw/lakes.geojson",
    params={"f": "json", "limit": 50},
    source="pygeoapi demo (Natural Earth lakes)",
    licence="public domain",
)
print(json.loads(path.with_suffix(".geojson.json").read_text())["sha256"][:16])
1e42a1b0f6b1f5cc

The sha256 is the useful part. Re-run the fetch in three months, compare the hash, and you know instantly whether the source changed — without diffing geometry.

Example 2 — deciding whether a cached copy is still good enough 

from datetime import datetime, timedelta, timezone
from pathlib import Path
import json

MAX_AGE = {
    "boundaries": timedelta(days=365),     # annual releases
    "osm": timedelta(days=7),              # changes constantly
    "imagery": timedelta(days=30),
}


def is_stale(sidecar, kind):
    meta = json.loads(Path(sidecar).read_text())
    age = datetime.now(timezone.utc) - datetime.fromisoformat(meta["retrieved"])
    return age > MAX_AGE[kind], age


stale, age = is_stale("data/raw/lakes.geojson.json", "boundaries")
print(f"age {age.days} days, stale={stale}")
age 0 days, stale=False

This is what turns "I think that file is recent" into a condition your pipeline can act on. Pair it with a real cache — see how to cache downloaded GIS data — so the check drives a refetch rather than a warning nobody reads.

Example 3 — an inventory of everything you have downloaded 

from pathlib import Path
import json
import pandas as pd

rows = []
for sidecar in Path("data/raw").rglob("*.json"):
    try:
        meta = json.loads(sidecar.read_text())
    except json.JSONDecodeError:
        continue
    if "retrieved" not in meta:
        continue                                    # not one of ours
    rows.append({
        "file": sidecar.name.removesuffix(".json"),
        "source": meta["source"],
        "licence": meta["licence"],
        "retrieved": meta["retrieved"][:10],
        "mb": round(meta["bytes"] / 1e6, 1),
    })

inventory = pd.DataFrame(rows).sort_values("retrieved")
print(inventory.to_string(index=False))
        file                              source        licence  retrieved   mb
lakes.geojson pygeoapi demo (Natural Earth lakes)  public domain 2026-08-26  0.1

When someone asks "can we publish this?", the answer is a table lookup instead of an archaeology project. The licence column is also the list you owe attribution to.

Explanation

Why the licence question is harder than it looks

Licences attach to data, and derived data is still data. Buffer an ODbL road network, dissolve the buffers, and publish the result: that polygon layer is a derived database under ODbL, and share-alike applies. Produce a map image from the same layer and, under the ODbL's "produced work" clause, it is not — you owe attribution but not share-alike.

The practical rule: images are usually safe, data usually is not. If your deliverable is a GeoPackage, licence compatibility is a design constraint from day one, not a step at the end.

Why "authoritative" does not mean "accurate"

An authoritative source is one with the standing to define the thing it records. A cadastre defines parcel boundaries because the legal parcel is the recorded one. That is a statement about authority, not about positional accuracy — an official boundary can be 5 m out and still be the correct boundary, because the record is the definition.

This matters when you mix sources. Overlaying an authoritative boundary on satellite imagery and finding they disagree does not mean the boundary is wrong. It usually means you have discovered the accuracy specification, which the metadata would have told you.

Why coverage is uneven in crowdsourced data, and how to check

OpenStreetMap has no coverage guarantee. Buildings are near-complete in some countries and near-absent in others; the same is true street by street. There is no flag for this — an empty query result looks exactly like "there is nothing there".

The check is comparison against something with a known denominator: count OSM buildings in a small area, compare with an official building or address count, and take the ratio as a completeness estimate for that area. It is crude, and it is far better than assuming.

Decision tree from the claim being made to the family of data source that can support it.
The claim picks the family. Browsing the portal first picks the claim, which is backwards.

Why the retrieval date belongs in the file name, not just the metadata

A sidecar answers "when was this fetched" for a file you still have. It does not stop you from overwriting last year's copy with this year's and losing the ability to reproduce last year's figures.

Writing boundaries_2026-08-26.gpkg and symlinking boundaries_latest.gpkg costs nothing and makes the pipeline reproducible against a point in time. This is the same argument as never overwriting batch outputs, applied to inputs.

Edge cases or notes

  • Portals lie about currency. A page saying "updated 2026" often means the page was updated. Trust the field inside the data or the HTTP Last-Modified header over the marketing copy.
  • A "national" dataset may exclude overseas territories, offshore installations and disputed areas. If your extent touches an edge case, check that specific area rather than the national totals.
  • Some APIs return a 200 with a partial result. The pygeoapi demo caps limit server-side: ask for 100000 and you receive 10, with no warning. Always compare what you received against the count the server reports — see GeoJSON downloaded from an API is empty or truncated.
  • Geocoding results are derived data. Coordinates you got from a geocoder inherit that geocoder's licence, which is frequently more restrictive than the addresses you sent it.
  • Imagery has a capture date, not a release date. A scene published last week may have been captured in March. For anything time-sensitive, filter on the capture property in the STAC catalogue.
  • Attribution has a place. Put it on the map, in the metadata of exported files, and in the repository README. One of the three will survive.

FAQ

What is the difference between authoritative and open data?

They are unrelated axes. Authoritative means the source has standing to define the thing (a census defines statistical geographies). Open means the licence permits reuse. Plenty of authoritative data is closed, and plenty of open data is unofficial.

Can I use OpenStreetMap in commercial work?

Yes. ODbL permits commercial use. What it restricts is redistribution of derived databases without the same licence, and it always requires attribution. Selling a map made from OSM is fine; selling a proprietary dataset derived from OSM is not.

How do I know if a dataset is complete enough?

There is no flag for it. Compare against a source with a known denominator for a small sample area — official building counts, census household counts, a road length total — and treat the ratio as an estimate for that area only. Completeness varies enormously by place.

Should I download the whole national file or query an API?

Query the API if your study area is fixed and small, download bulk if you will slice it many ways or need to work offline. The deciding factor is usually how often the source changes: bulk files get stale silently, API results do not.

Where does the retrieval date go?

In three places: the file name, a sidecar JSON, and the metadata of anything you derive from it. Redundant on purpose — each survives a different kind of accident.

Is a basemap a data source?

No. Tiles are rendered pixels; the features that made them are not in the file. Use them for context and take measurements from the vector data underneath. See how to add a basemap.

What if the licence is unclear?

Treat unclear as closed until someone tells you otherwise, and ask the publisher — most respond. An unclear licence discovered after publication is a much worse conversation than one asked about beforehand.