🔧 Fix Privacy & sensitive locations Census & demographics Geoprivacy & Anonymisation · step 18 of 20 Updated

Suppressed cells can be recovered from the totals

Problem statement

The table was suppressed correctly by the rule everyone uses: any cell below five is blanked. Then a user noticed that one row published four of its five categories and its total, and subtracted.

Suppression hides values from the page, not from the arithmetic. A blanked cell in a row or column with exactly one hole and a published margin has a single solution; a table with several holes can still be solved if the holes are arranged badly. On a real table — 130 cells of 250 m central London crime by 14 categories — 6 of the 130 rows had exactly one suppressed value and were therefore fully recoverable.

Quick answer

Find the recoverable cells, then suppress a second value in each affected row and column until none remain:

import numpy as np

v = table.values
nonempty = v > 0
sup = nonempty & (v < 5)                       # primary suppression

while True:
    changed = False
    for i in range(v.shape[0]):
        if sup[i].sum() == 1:
            cand = np.flatnonzero(nonempty[i] & ~sup[i])
            if len(cand):
                sup[i, cand[v[i, cand].argmin()]] = True
                changed = True
    for j in range(v.shape[1]):
        if sup[:, j].sum() == 1:
            cand = np.flatnonzero(nonempty[:, j] & ~sup[:, j])
            if len(cand):
                sup[cand[v[cand, j].argmin()], j] = True
                changed = True
    if not changed:
        break

On the London table this stabilised in two passes and cost three extra cells: 643 primary suppressions became 646, hiding 1,361 records instead of 1,343.

Triage of ways a suppressed cell is recoverable and the fix for each.
The row total is the obvious channel; the time series and the second geography are the ones that get missed.

Step-by-step solution

1. Enumerate every published margin

Row totals, column totals, the grand total, subtotals, "all categories" rows, the map legend's maximum, and any figure quoted in the accompanying text. Each is an equation.

2. Find rows and columns with exactly one hole 

sup = (table > 0) & (table < 5)
print("rows with one hole:", int((sup.sum(axis=1) == 1).sum()))
print("cols with one hole:", int((sup.sum(axis=0) == 1).sum()))
rows with one hole: 6
cols with one hole: 0

Six recoverable cells out of 643 suppressions. Small, and completely defeating the purpose for those six.

3. Suppress the cheapest additional cell in each

Choosing the smallest remaining value minimises the information lost, because a small value is the least useful published number in the row.

4. Iterate — one fix creates another

Suppressing a second cell in a row can leave that cell's column with exactly one hole. Loop until a pass makes no change. Two passes sufficed here; a denser table takes more.

5. Handle rows that cannot be repaired

Three of the 130 rows had only one non-empty cell, so there is no second value to hide. The only correct responses are to suppress the row total as well, or to merge the row into a neighbouring area. Leaving it is publishing the value.

6. Check the time dimension

A cell suppressed in one month and published in another, with a published annual total, is recoverable by subtraction across the series. Suppression decisions must be made jointly over every release, not per release.

7. Check the second geography

If the same population is published at two nested geographies — wards and districts — the difference between a district and the sum of its published wards bounds the suppressed wards. Overlapping non-nested geographies are worse, because they produce a system of equations.

8. Consider rounding instead of holes

Controlled rounding publishes a number in every cell and removes the arithmetic entirely. It is usually the better answer for open data, and it is what several national statistics offices now do.

Table showing a row with four published values, one blank and a published total, with the blank solved by subtraction.
Four knowns and a total make the fifth value arithmetic, not a secret.

Code examples

Example 1 — an attacker's solver 

import numpy as np

def recover(published, row_totals, col_totals):
    """Return cells that are determined exactly by the published margins."""
    known = ~np.isnan(published)
    solved = {}
    changed = True
    while changed:
        changed = False
        for i, total in enumerate(row_totals):
            missing = np.flatnonzero(~known[i])
            if len(missing) == 1:
                j = missing[0]
                published[i, j] = total - np.nansum(published[i])
                known[i, j] = True
                solved[(i, j)] = published[i, j]
                changed = True
        for j, total in enumerate(col_totals):
            missing = np.flatnonzero(~known[:, j])
            if len(missing) == 1:
                i = missing[0]
                published[i, j] = total - np.nansum(published[:, j])
                known[i, j] = True
                solved[(i, j)] = published[i, j]
                changed = True
    return solved

solved = recover(published.copy(), row_totals, col_totals)
print(f"{len(solved)} suppressed cells recovered exactly")

Run this against your own release. If it returns anything, the suppression is incomplete.

Example 2 — bounds, not just exact solutions 

from scipy.optimize import linprog

def feasible_range(published, row_totals, col_totals, target):
    """Minimum and maximum a suppressed cell can take, given the margins."""
    idx = [(i, j) for i in range(published.shape[0])
           for j in range(published.shape[1]) if np.isnan(published[i, j])]
    n = len(idx)
    A, b = [], []
    for i, total in enumerate(row_totals):
        row = [1 if a == i else 0 for a, _ in idx]
        if any(row):
            A.append(row); b.append(total - np.nansum(published[i]))
    for j, total in enumerate(col_totals):
        col = [1 if b_ == j else 0 for _, b_ in idx]
        if any(col):
            A.append(col); b.append(total - np.nansum(published[:, j]))
    c = np.zeros(n); c[idx.index(target)] = 1
    lo = linprog(c, A_eq=A, b_eq=b, bounds=[(0, None)] * n)
    hi = linprog(-c, A_eq=A, b_eq=b, bounds=[(0, None)] * n)
    return lo.fun, -hi.fun

A cell whose feasible range is [2, 2] is published. A cell whose range is [0, 4] is protected. Statistical agencies size the suppression pattern by requiring every range to be wider than a protection level — which is what the heuristic loop approximates.

Example 3 — what the repair costs 

before = ((table > 0) & (table < 5)).values
after = suppress(table, 5)[1]

print(f"primary:      {before.sum():,} cells, {table.values[before].sum():,} records")
print(f"complementary:{after.sum():,} cells, {table.values[after].sum():,} records")
print(f"extra cost:   {after.sum() - before.sum()} cells, "
      f"{table.values[after].sum() - table.values[before].sum()} records")
primary:      643 cells, 1,343 records
complementary:646 cells, 1,361 records
extra cost:   3 cells, 18 records

Three cells. The fix is nearly free on a wide table, which is why there is no excuse for skipping it.

Explanation

Why the heuristic is not a guarantee

Requiring at least two holes per row and column makes each single equation underdetermined. It does not prove the system is: a cell with two holes in its row can still be pinned if the other hole is fixed by its own column. Proper cell suppression solves a linear programme that gives every suppressed cell a feasible interval of a chosen width. For an open-data table the heuristic is usually adequate; for a release under attack, use a solver and check the intervals.

Why time series are the commonest real failure

Nobody re-derives the suppression pattern across twelve monthly releases and an annual total, because the twelve releases are produced by twelve pipeline runs that know nothing about each other. The suppression state has to be stored and reused, exactly like a mask.

Why nested geographies leak

Publishing wards and districts publishes a set of equations linking them. If a district has five wards and one ward is suppressed, the district total minus the four published wards is the suppressed value. Two geographies are two chances to get the pattern wrong.

Why rounding sidesteps the whole problem

With controlled rounding there are no holes, so there are no equations to solve; every cell carries an approximate number and the margins are made consistent by construction. The trade is that a rounded value bounds the truth, so the base has to be large enough that the bound is not itself disclosive.

Bars comparing primary suppression of 643 cells hiding 1,343 records with complementary suppression of 646 cells hiding 1,361 records.
On a wide table the repair is nearly free, which is why skipping it is indefensible.

Edge cases or notes

  • The grand total is a margin. So is a number quoted in a press release.
  • Percentages publish the denominator. A rate and a count give the base.
  • Zeros are not holes. A published zero is a known value and helps the solver.
  • Store the suppression mask. Next release must reuse it, not recompute it.
  • Maps have margins too. A legend maximum publishes the largest cell.
  • Subtotals multiply the equations. Every subtotal is another constraint.
  • Check the published CSV, not the dashboard. Downloads usually carry more than the map does.
  • Document the rule. Users who know a blank means "<5" will not read a blank as zero.

FAQ

Why is suppressing small cells not enough?

Because a blanked cell in a row with a published total is a subtraction. Six rows in the test table had exactly one hole and could be solved exactly.

What is complementary suppression?

Blanking additional cells so no suppressed value can be derived from a published margin. In practice: never leave exactly one hole in a row or column.

How much does complementary suppression cost?

On a wide table, very little. Repairing the London table cost three extra cells and eighteen extra hidden records.

What if a row has only one non-empty cell?

There is nothing to pair it with. Suppress the row total, or merge the row into a neighbouring area.

Do I need a solver, or is the heuristic enough?

The heuristic covers most open-data cases. If the table has subtotals, several margins, or a determined adversary, compute each suppressed cell's feasible interval with a linear programme.

How do time series leak suppressed cells?

A cell suppressed in one period and published in another, with a published total across periods, is recoverable by subtraction. Store the suppression mask and reuse it.