How to Download Census Tables from an API in Python
On this page
Problem statement
Most tutorials for US census data start with one request to the Census Data API:
import requests
r = requests.get("https://api.census.gov/data/2023/acs/acs5",
params={"get": "NAME,B01003_001E", "for": "county:*", "in": "state:10"})
r.raise_for_status()
rows = r.json()Run without an API key on 11 September 2026, and again on 28 September, that code did not return data. The API answered with a 302 redirect to missing_key.html. requests followed it and received an HTML page titled "Missing Key" with status 200, so raise_for_status() passed, and the failure only appeared one line later:
r = requests.get("https://api.census.gov/data/2023/acs/acs5",
params={"get": "NAME,B01003_001E", "for": "county:*", "in": "state:10"}, timeout=60)
print(r.status_code, [h.status_code for h in r.history], r.url)
try:
r.json()
except requests.exceptions.JSONDecodeError as error:
print(f"JSONDecodeError: {error}")200 [302] https://api.census.gov/data/missing_key.html
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The metadata endpoints still answered without a key — variable definitions, table groups and the list of supported geographies all returned JSON. And the Census Bureau publishes the same tables as bulk files that need no key at all: the whole country's tract populations downloaded and parsed in 1.7 s.
This guide covers both routes: the API done properly, with a key and with failures that name themselves, and the bulk summary files for when you want every geography at once.
Quick answer
Get a free key from the Census Bureau's key signup page, put it in an environment variable, and make the missing-key case raise a clear error:
import os
import requests
import pandas as pd
params = {"get": "NAME,B01003_001E,B01003_001M", "for": "tract:*", "in": "state:10",
"key": os.environ["CENSUS_API_KEY"]}
r = requests.get("https://api.census.gov/data/2023/acs/acs5", params=params,
timeout=60, allow_redirects=False)
if r.is_redirect:
raise RuntimeError(f"Census API redirected to {r.headers['Location']}")
r.raise_for_status()
rows = r.json()
frame = pd.DataFrame(rows[1:], columns=rows[0]) # every value arrives as textWithout a key, or when you need a table for every tract in the country, read the table-based summary file instead — no key, one request:
url = ("https://www2.census.gov/programs-surveys/acs/summary_file/2023/"
"table-based-SF/data/5YRData/acsdt5y2023-b01003.dat")
table = pd.read_csv(url, sep="|", dtype=str)
tracts = table[table.GEO_ID.str.startswith("1400000US")]
print(len(tracts)) # 8538185381
Keep the key safe, find variables, request by geography
1. Store the key outside the code
The key is free and arrives by email. Keep it in an environment variable or a secrets manager, never in a notebook or a repository:
export CENSUS_API_KEY="your-key-here"Every data request needs key= in its parameters; a request without one gets the redirect. Activate the key from the link in the confirmation email before using it.
2. Discover variables without spending requests
Variable names are not guessable, and the metadata endpoints need no key. A table's group listing returns every variable in it:
group = requests.get("https://api.census.gov/data/2023/acs/acs5/groups/B17001.json",
timeout=60).json()["variables"]
estimates = sorted(v for v in group if v.endswith("E") and v != "NAME")
margins = sorted(v for v in group if v.endswith("M"))
print(len(group), len(estimates), len(margins))
print(f"B17001_002E: {group['B17001_002E']['label']} | {group['B17001_002E']['predicateType']}")238 59 59
B17001_002E: Estimate!!Total:!!Income in the past 12 months below poverty level: | int
B17001, poverty status by sex and age, has 59 estimates, 59 margins of error and 118 annotation variables, plus GEO_ID and NAME. Filter the names carefully: a suffix test for E also matches NAME, which is how a first attempt counted 60 estimates. Each variable also has its own record, with a readable label:
3. Build the geography from what the API requires
for names the level you want and in names its parents. The API's geography.json lists the 65 supported levels and what each requires:
county requires state wildcard allowed for state
tract requires state, county wildcard allowed for county
block group requires state, county, tract wildcard allowed for county, tract
zip code tabulation area requires nothingSo every tract in Delaware is for=tract:*&in=state:10, and every tract in the country is 52 calls — one each for the 50 states, the District of Columbia and Puerto Rico. That is the point at which the bulk file becomes the better route.
4. Turn the response into typed columns
The response is a JSON array of arrays, with a header row first. Every value is a string, including the numbers, and the geography comes back as separate state, county and tract columns:
frame["GEOID"] = frame["state"] + frame["county"] + frame["tract"]
for col in ("B01003_001E", "B01003_001M"):
frame[col] = pd.to_numeric(frame[col])Concatenate the geography columns as text, which keeps their leading zeros, and convert only the value columns to numbers. Then replace the annotation sentinels — values such as −666666666 — before any arithmetic.
5. Split long variable lists across requests
The API accepts at most 50 variables per request, and NAME counts. B17001's 59 estimates and 59 margins need three requests at 48 variables each. Merge the pieces on the geography columns, never by row position.
6. Use the bulk summary file for whole-country tables
The table-based summary file holds one pipe-delimited file per table, with every geography level in it, no key required:
B01003, tracts, over HTTP 85,381 rows in 1.7 s
B17001, four columns, tracts 85,381 rows in 3.5 s (file 118.4 MB)
B19013, counties 3,222 rows in 1.6 sThe B01003 file holds 616,634 rows across 65 summary levels (Example 4). The naming differs from the API: the summary file calls the estimate B01003_E001 and the margin B01003_M001, where the API says B01003_001E and B01003_001M. Names for each GEO_ID are in a separate geography file, Geos20235YR.txt, with 616,701 rows.
7. Expect the sentinels in either route
Both routes carry the ACS annotation values. In the tract-level median household income table B19013, 1,547 estimates were −666666666, and the margins held 1,547 values of −222222222 and 492 of −333333333. A mean over that column without cleaning it is meaningless.
8. Outside the US
The same pattern works elsewhere, and neither of these needed a key when measured:
- ONS census data for England and Wales is served by Nomis. A CSV request for LSOA usual residents returned in 8.2 s, keyed by codes such as
E01011954— and returned exactly 25,000 rows, the cap on one keyless Nomis request, fewer than the 33,755 LSOAs in England alone. Page through the rest withrecordoffset, or check the row count against the geography you asked for. - Eurostat serves JSON-stat. A request for the 2023 population of
DE300(Berlin) andFR101(Paris) returned 3,632,853 and 2,092,813 in under a second.
API calls that fail with a reason
The data used on this page
Nothing is downloaded in advance. The requests go to the Census Data API without a key, to show exactly what a keyless call returns, and to the key-free metadata endpoints and ACS table-based summary files. The calls that need a key are shown but were not re-run for this page, because this site holds no key; everything else below ran on 28 September 2026.
import os
import time
import pandas as pd
import requestsExample 1 — an API call that fails with a reason
import os
import pandas as pd
import requests
API = "https://api.census.gov/data"
class CensusKeyError(RuntimeError):
pass
def census_get(dataset, variables, for_, in_=None, key=None, timeout=60):
"""One Census Data API call. Returns a DataFrame of strings."""
key = key or os.environ.get("CENSUS_API_KEY")
params = {"get": ",".join(variables), "for": for_}
if in_:
params["in"] = in_
if key:
params["key"] = key
r = requests.get(f"{API}/{dataset}", params=params, timeout=timeout,
allow_redirects=False)
if r.is_redirect and "key" in r.headers.get("Location", ""):
raise CensusKeyError(f"{r.status_code} -> {r.headers['Location']}: "
"set CENSUS_API_KEY (free, from api.census.gov/data/key_signup.html)")
r.raise_for_status()
rows = r.json()
return pd.DataFrame(rows[1:], columns=rows[0])
import time
start = time.perf_counter()
try:
census_get("2023/acs/acs5", ["NAME", "B01003_001E"], "county:*", "state:10", key="")
except CensusKeyError as error:
print(f"CensusKeyError: {error} ({time.perf_counter() - start:.1f} s)")Without a key:
04
The error arrives in well under a second. allow_redirects=False is the important argument: with the default, the redirect is followed and the only symptom is an HTML body that is not JSON.
Example 2 — a whole table in batches under the variable limit
def chunks(items, size):
for i in range(0, len(items), size):
yield items[i:i + size]
def census_group(dataset, group, for_, in_=None, per_call=48):
"""Every estimate and margin in a table group, merged on the geography columns."""
meta = requests.get(f"{API}/{dataset}/groups/{group}.json", timeout=60).json()["variables"]
wanted = sorted(v for v in meta if v.startswith(group) and v[-1] in "EM")
geo_cols = None
merged = None
for batch in chunks(wanted, per_call):
part = census_get(dataset, batch, for_, in_)
geo_cols = [c for c in part.columns if c not in batch]
part[batch] = part[batch].apply(pd.to_numeric)
merged = part if merged is None else merged.merge(part, on=geo_cols, validate="one_to_one")
print(f"{group}: {len(wanted)} variables in {-(-len(wanted) // per_call)} calls, {len(merged):,} rows")
return mergedFor B17001 the variable list has 118 entries — the startswith(group) test excludes NAME and GEO_ID, and the E/M test excludes the annotations — so the table arrives in three calls. The merge uses the geography columns and validate, so a batch that returned a different set of rows raises instead of misaligning values.
Example 3 — the keyless bulk route
def acs_summary_table(table, level_prefix="1400000US", year=2023, columns=None):
"""Whole-country ACS 5-year table from the bulk summary file; no key needed."""
url = (f"https://www2.census.gov/programs-surveys/acs/summary_file/{year}/"
f"table-based-SF/data/5YRData/acsdt5y{year}-{table.lower()}.dat")
usecols = None if columns is None else ["GEO_ID"] + columns
frame = pd.read_csv(url, sep="|", dtype=str, usecols=usecols)
frame = frame[frame.GEO_ID.str.startswith(level_prefix)].copy()
frame.insert(0, "GEOID", frame.GEO_ID.str[len(level_prefix):])
value_cols = [c for c in frame.columns if c not in ("GEO_ID", "GEOID")]
frame[value_cols] = frame[value_cols].apply(pd.to_numeric)
return frame.drop(columns="GEO_ID")
start = time.perf_counter()
b01003 = acs_summary_table("B01003")
print(f"B01003 tracts over HTTP: {len(b01003):,} rows in {time.perf_counter() - start:.1f} s")
print(b01003.head(2).to_string(index=False))B01003 tracts over HTTP: 85,381 rows in 1.8 s
GEOID B01003_E001 B01003_M001
01001020100 1840 358
01001020200 2017 316
dtype=str keeps the identifiers intact before the value columns are converted. The same URL pattern answered for the 2022 and 2024 releases. Download the file once and read it locally when you use it repeatedly — B17001 alone is 118.4 MB.
Example 4 — the other measurements on this page
geo = requests.get("https://api.census.gov/data/2023/acs/acs5/geography.json", timeout=60).json()["fips"]
print(f"{len(geo)} geography levels")
seen = set()
for level in geo:
if level["name"] in ("county", "tract", "block group", "zip code tabulation area") and level["name"] not in seen:
seen.add(level["name"])
print(f"{level['name']:26} requires {', '.join(level.get('requires', [])) or 'nothing':26} "
f"wildcard allowed for {', '.join(level.get('wildcard', [])) or '-'}")
for table, prefix, cols in (("B17001", "1400000US", ["B17001_E001", "B17001_M001", "B17001_E002", "B17001_M002"]),
("B19013", "0500000US", None)):
start = time.perf_counter()
t = acs_summary_table(table, prefix, columns=cols)
print(f"{table} over HTTP: {len(t):,} rows in {time.perf_counter() - start:.1f} s")
full = pd.read_csv("https://www2.census.gov/programs-surveys/acs/summary_file/2023/table-based-SF/data/5YRData/"
"acsdt5y2023-b01003.dat", sep="|", dtype=str, usecols=["GEO_ID"])
print(f"B01003 file: {len(full):,} rows, {full.GEO_ID.str[:3].nunique()} summary levels")
start = time.perf_counter()
nomis = pd.read_csv("https://www.nomisweb.co.uk/api/v01/dataset/NM_2021_1.data.csv?date=latest"
"&geography=TYPE151&c2021_restype_3=0&measures=20100&select=geography_code,obs_value")
print(f"Nomis LSOA usual residents: {len(nomis):,} rows in {time.perf_counter() - start:.1f} s, "
f"first code {nomis.GEOGRAPHY_CODE.iloc[0]}")
start = time.perf_counter()
es = requests.get("https://ec.europa.eu/eurostat/api/dissemination/statistics/1.0/data/demo_r_pjanaggr3",
params={"geo": ["DE300", "FR101"], "time": "2023", "sex": "T", "age": "TOTAL"}, timeout=60).json()
print(f"Eurostat 2023 population: {dict(zip(es['dimension']['geo']['category']['index'], es['value'].values()))} "
f"in {time.perf_counter() - start:.1f} s")65 geography levels
county requires state wildcard allowed for state
tract requires state, county wildcard allowed for county
block group requires state, county, tract wildcard allowed for county, tract
zip code tabulation area requires nothing wildcard allowed for -
B17001 over HTTP: 85,381 rows in 3.5 s
B19013 over HTTP: 3,222 rows in 1.6 s
B01003 file: 616,634 rows, 65 summary levels
Nomis LSOA usual residents: 25,000 rows in 8.2 s, first code E01011954
Eurostat 2023 population: {'DE300': 3632853, 'FR101': 2092813} in 0.2 s
Failures inside a 200, and strings everywhere
Why the failure hides in a 200
HTTP status codes describe the last response, and requests follows redirects unless told not to. The 302 that meant "no key" is in r.history; the response you hold is a successful delivery of an HTML page. So every check that looks at r.status_code or calls raise_for_status() passes, and the code fails on the JSON parse, with an error that mentions neither census data nor keys.
Turning redirects off for data calls makes the real response visible. It costs nothing, because a data endpoint has no legitimate reason to redirect.
Why metadata is free and data is not
The metadata endpoints — variables, groups, geography.json and the dataset catalogue — describe what exists and are cheap to serve. Data calls are what a key meters. That split is useful: you can discover variables, check geography requirements and validate a request plan before spending any authenticated calls.
Why every value is a string
The API returns one JSON format for every dataset, including text fields such as NAME and codes with leading zeros. Returning everything as strings keeps codes such as county 001 intact; it also means a numeric column is text until you convert it. Convert values explicitly, and build GEOID by concatenating the text geography columns.
When the bulk file is the better tool
An API call is right for a few variables in a few places. A national table at tract level would take one call per state for each batch of 48 variables, against one download that already contains every summary level. The bulk file also removes the key and rate limits from a pipeline entirely, at the price of downloading whole tables.
Unactivated keys and annotation variables
- An unactivated key behaves like no key. The confirmation email contains an activation link; until it is followed, requests redirect.
NAMEends in E. Filter variable lists by table prefix, not only by suffix.- Annotation variables end in EA and MA. They explain a sentinel value; request them only when you need the reason.
- The summary file and the API name variables differently:
B01003_E001againstB01003_001E. - Filter summary levels with the full prefix.
1400000USgives 85,381 tracts; shorter prefixes include partial geographies. - Vintages are directories. The catalogue listed 5-year ACS vintages through 2024; change the year in the path, and match the boundary file year.
- Timeouts happen. One metadata request timed out after 60 s during testing; set a timeout and retry idempotent GETs.
- Nomis and Eurostat use their own geography codes —
E01…LSOA codes and NUTS codes such asDE300— which do not need zero padding.
Internal links
- Census geographies explained: blocks, tracts, output areas and why they nest — what the for and in levels are
- Census identifiers explained: GEOIDs, codes and the leading zero problem — building GEOID from the geography columns
- How to join a census table to its boundaries without losing rows — the next step
- Fixing census API values like −666666666 and other sentinels — cleaning what arrives
- Margins of error explained: why survey estimates need their uncertainty — why to download the M columns too
- How to download census boundaries in Python — the polygons for these tables
- How to handle credentials and secrets in an automated GIS job — keeping the key out of code
- How to add retries and timeouts to an automated GIS job — for the requests that time out
FAQ
Do I need an API key for the Census Data API?
For data requests, yes. Measured on 11 September 2026, a keyless request was redirected to missing_key.html. Metadata endpoints such as variables and geography.json still worked without one.
Why does my Census API request return JSONDecodeError?
The API probably redirected to its missing-key page, and requests followed the redirect to an HTML page with status 200. Pass allow_redirects=False to see the redirect, and add a key.
How many variables can I request at once?
Up to 50 per request, including NAME. Split larger tables into batches and merge the results on the geography columns.
How do I get data for every census tract in the US?
Either one API call per state, since tracts require a state, or one download of the table-based summary file, which read all 85,381 tracts of B01003 in 1.7 seconds with no key.
Why are all the numbers strings?
The API returns every value as text so that codes keep their leading zeros. Convert the value columns with pd.to_numeric and concatenate the geography columns into a GEOID as text.
Is there an equivalent for UK or EU census data?
Nomis serves ONS census tables as CSV, and Eurostat serves regional statistics as JSON-stat. Both answered without a key when tested.