Margins of Error Explained: Why Survey Estimates Need Their Uncertainty #

Problem statement #

Every number in the American Community Survey comes with a second number. B17001_E002 is the estimated count of people below the poverty line; B17001_M002 is its margin of error. Most maps, dashboards and spreadsheets use the first column and drop the second.

At county level that is a forgivable shortcut. At tract level it is not. Measured on the 2023 ACS 5-year poverty table:

• The median tract estimate of people in poverty has a coefficient of variation of 34.1%, against 9.9% for the median county.
• Only 212 of 85,381 tract estimates count as highly reliable by the common 12% rule; 27,251 are low reliability.
• Only 45.7% of tracts have a poverty rate statistically different from their county's — so for more than half the tracts on a map, the colour difference from the county average is not supported by the data.

A margin of error is not an optional extra column. It is the part of the estimate that says how much of the map to believe.

Quick answer #

ACS margins of error are 90% margins. Convert them to a standard error and a coefficient of variation before deciding anything:

se = moe / 1.645                    # standard error
cv = se / estimate                  # coefficient of variation
low, high = estimate - moe, estimate + moe

A tract with 100 people below poverty and a margin of 68 has a standard error of 41.3, a CV of 41.3%, and a 90% interval from 32 to 168. That estimate cannot tell 50 people from 150.

Use the CV to flag reliability, and test differences with both margins before claiming that two places differ:

different = abs(est1 - est2) / ((moe1 / 1.645) ** 2 + (moe2 / 1.645) ** 2) ** 0.5 > 1.645

Step-by-step solution #

1. Know what the margin measures #

The ACS is a survey of a sample of addresses, pooled over five years for the 5-year release. Each estimate would come out differently with a different sample, and the margin of error describes that sampling variation. At 90% confidence, the interval estimate ± MOE is expected to contain the value a complete count would give in nine samples out of ten.

The margin covers sampling error only. Non-response, coding mistakes and the gap between a five-year average and a single year are not in it.

2. Convert the margin to the scale you need #

standard error          SE  = MOE / 1.645
coefficient of variation CV = SE / estimate
95% margin              MOE95 = MOE × 1.96 / 1.645 = MOE × 1.1915

Other agencies publish 95% intervals, so convert before comparing an ACS margin with one from elsewhere.

3. Judge reliability by the CV, not the margin #

A margin of 68 is large for an estimate of 100 and trivial for 100,000. The CV removes the scale. A widely used convention treats a CV up to 12% as high reliability, 12–40% as medium and above 40% as low:

estimate: people below poverty    high      medium     low      zero estimate
tracts                              212     56,598   27,251          1,320
counties                          2,059      1,138       25              —

The median CV was 34.1% for tracts and 9.9% for counties. A third of tract estimates (32.4%) had a CV above 40%, and 5.9% had a margin at least as large as the estimate itself.

4. Expect reliability to depend on the variable and the level #

Totals are more reliable than subgroups, and larger areas more than smaller ones:

median CV                         county    tract    block group
people below poverty                9.9%    34.1%    not published
median household income             4.1%    12.4%       20.0%
total population                      —      8.4%       18.6%

Median household income is published for 221,369 block groups, and a quarter of them had a CV above 30%. The poverty table is not published for block groups at all.

5. Recognise estimates that have no margin #

County total population is missing from the county row above for a reason. Its margin is the sentinel −555555555 in 3,090 of 3,222 counties and in all 52 states. That code means the estimate is controlled to the Census Bureau's official population estimates, so a sampling margin does not apply. The 132 counties with a real margin are those that are not controlled this way, with populations from 43 to 164,632.

Treat −555555555 as "no sampling error to report", never as a number.

6. Do not treat a zero as certain #

1,320 tracts have an estimate of zero people below poverty, and their margins are not zero — the median is 13. The survey found nobody, which is consistent with a small number who were not sampled. Map zeros as "fewer than about the margin", not as a firm absence.

7. Test differences before describing them #

Two estimates differ at 90% confidence when their difference is larger than 1.645 standard errors of the difference:

z = abs(p1 - p2) / ((moe1 / 1.645) ** 2 + (moe2 / 1.645) ** 2) ** 0.5

Applied to every tract with at least 100 people in the poverty universe, 45.7% had a poverty rate significantly different from their county's, and for 53.3% the tract's own 90% interval contained the county rate. A choropleth colours every one of those tracts differently from the county average anyway.

8. Be careful with rankings #

Ranks magnify uncertainty. In Cook County, Illinois, 1,328 tracts have a poverty universe of at least 100. The 90% interval of the median-ranked tract overlapped the intervals of 1,060 of them — 80%. A ranked list of tracts by poverty rate is mostly an ordering of noise in its middle.

Code examples #

Example 1 — reliability for a whole column #

import numpy as np
import pandas as pd

Z90 = 1.645
SENTINELS = [-222222222, -333333333, -555555555, -666666666, -888888888, -999999999]


def reliability(estimate, moe, bands=(0.12, 0.40)):
    """Standard error, coefficient of variation and a reliability class for ACS estimates."""
    est = estimate.where(~estimate.isin(SENTINELS))
    m = moe.where(~moe.isin(SENTINELS))
    se = m / Z90
    cv = se / est.where(est > 0)
    label = pd.cut(cv, [0, bands[0], bands[1], np.inf], labels=["high", "medium", "low"], include_lowest=True)
    label = label.cat.add_categories(["no MOE", "zero estimate"])
    label[m.isna()] = "no MOE"
    label[(est == 0) & m.notna()] = "zero estimate"
    print(label.value_counts().reindex(["high", "medium", "low", "zero estimate", "no MOE"]).to_dict())
    return pd.DataFrame({"estimate": est, "moe": m, "se": se, "cv": cv, "reliability": label})

tract people below poverty:
{'high': 212, 'medium': 56598, 'low': 27251, 'zero estimate': 1320, 'no MOE': 0}
county people below poverty:
{'high': 2059, 'medium': 1138, 'low': 25, 'zero estimate': 0, 'no MOE': 0}
county total population:
{'high': 121, 'medium': 10, 'low': 1, 'zero estimate': 0, 'no MOE': 3090}

Sentinels become missing before any arithmetic, so a −555555555 margin cannot produce a negative CV. Keep the reliability column next to the estimate all the way to the map.

Example 2 — the margin of a rate #

def proportion_moe(num, moe_num, den, moe_den):
    """MOE of num/den when num is a subset of den; falls back to the ratio formula."""
    p = num / den
    radicand = moe_num ** 2 - p ** 2 * moe_den ** 2
    ratio = moe_num ** 2 + p ** 2 * moe_den ** 2
    used_ratio = radicand < 0
    moe = np.sqrt(np.where(used_ratio, ratio, radicand)) / den
    print(f"{int(used_ratio.sum())} of {len(p):,} used the ratio formula")
    return p, pd.Series(moe, index=num.index)

95 of 84,088 used the ratio formula
0 of 3,222 used the ratio formula
median tract rate MOE pp 5.6 county 2.3

This is the Census Bureau's approximation for a proportion whose numerator is part of its denominator. When the quantity under the square root is negative — 95 tracts here — the documented fallback is the ratio formula, which adds the terms instead. The median tract poverty rate carries a margin of 5.6 percentage points.

Example 3 — which differences are real #

def differ(est1, moe1, est2, moe2, z=Z90):
    """Are two independent estimates different at the confidence level of their MOEs?"""
    stat = (est1 - est2).abs() / np.sqrt((moe1 / Z90) ** 2 + (moe2 / Z90) ** 2)
    return stat > z

tracts different from their county: 45.7% of 84,088

The test assumes the two estimates are independent. A tract and the county containing it are not, because the tract's sample is part of the county's, so the test is conservative for this comparison. Pass z=1.96 to test at 95% confidence while keeping the published 90% margins as input.

Explanation #

Why tract margins are so wide #

The precision of a survey estimate depends mostly on how many responses sit behind it, not on the population it describes. A county's estimate draws on thousands of sampled households; a tract of 4,000 people draws on a small fraction of that. Subgroups are thinner still: people below poverty in a tract are a subset of a subset. That is why the median CV rises from 9.9% for county poverty counts to 34.1% for tracts, and why block-group poverty is not published at all.

Why the CV is the right yardstick #

A margin in absolute units cannot be compared across estimates of different sizes. Dividing the standard error by the estimate expresses the uncertainty as a share of the value, so a 12% threshold means the same thing for 100 people and for 100,000. It fails only near zero, where the CV grows without limit — which is why zero and near-zero estimates need their own treatment.

Why overlapping intervals are not a test #

Two 90% intervals can overlap and the estimates still be significantly different, because the standard error of a difference is smaller than the sum of the two standard errors. Comparing intervals by eye is conservative; the z-test in Example 3 is the correct check, and it is one line.

Why controlled estimates have no margin #

The ACS weights its responses so that county totals of population and housing match the Census Bureau's separate population estimates. Those totals are therefore fixed by the weighting rather than estimated from the sample, and the survey has no sampling error to report for them. The sentinel −555555555 says exactly that. Characteristics within those totals — poverty, income, age groups — still have sampling error and still carry margins.

Edge cases or notes #

• Margins are published at 90%. Multiply by 1.1915 for 95%; divide by 1.645 for a standard error.
• Negative sentinel margins are codes, not values. −555555555 means controlled; others mean the margin could not be calculated or the estimate is suppressed.
• Zero estimates have positive margins. The median for tracts with no one below poverty was 13.
• Margins cannot be summed. Aggregating estimates needs the root-sum-of-squares approximation, which has its own biases.
• The reliability bands are a convention. 12% and 40% are widely used, not official; state the bands you apply.
• A rate's margin is not the count's margin divided by the denominator. Use the proportion formula, with the ratio fallback.
• 5-year estimates are period averages. They describe the five years pooled, not the final year.
• Other surveys publish other confidence levels. Convert to a common level before comparing.

Internal links #

• How to aggregate survey estimates and their margins of error — combining tracts without losing the uncertainty
• Small numbers explained: why rates in small areas jump around — the other reason small-area rates mislead
• Counts, rates and densities: what a demographic map should show — which estimate to map in the first place
• Fixing census API values like −666666666 and other sentinels — every sentinel code and what it means
• How to download census tables from an API in Python — getting the M columns along with the E columns
• Fixing a rate map dominated by tiny, nearly empty areas — showing reliability on the map
• Choropleth classification explained — class breaks that respect uncertainty
• Spatial data quality: the six dimensions that matter — accuracy as one dimension among several

FAQ #

What confidence level do ACS margins of error use? #

90%. Divide the margin by 1.645 to get a standard error, or multiply it by 1.1915 to get a 95% margin.

How do I know whether an ACS estimate is reliable? #

Compute its coefficient of variation: the margin divided by 1.645, divided by the estimate. A common convention treats up to 12% as high reliability and above 40% as low; the median tract poverty count was 34.1%.

Why does my county population have a margin of −555555555? #

The estimate is controlled to the Census Bureau's official population estimates, so there is no sampling error to report. That applied to 3,090 of 3,222 counties in the 2023 5-year release.

Can I say one tract has a higher poverty rate than another? #

Only after a significance test using both margins. For 45.7% of tracts, the poverty rate was significantly different from the county rate; for the rest, the data cannot support the difference.

Why does an estimate of zero have a margin of error? #

Because the survey samples households. Finding nobody in the sample is consistent with a few people in the population; the median margin for tracts with a zero poverty count was 13.

Should I use block groups for more detail? #

Only for variables published at that level and with their margins checked. Median household income at block-group level had a median CV of 20.0%, and the poverty table is not published for block groups.