Repair, Reject or Flag? Choosing What Cleaning Should Do

Problem statement

The cleaning function works. Nobody can say what it did. Here it is on a real register — every food business Leeds City Council inspects, 7,454 rows from the Food Standards Agency (the setup is under The data used on this page):

def clean_naive(df):
    df = df[df.geometry.notna()]
    df = df.drop_duplicates(subset=["name", "postcode"])
    df = df.assign(name=df["name"].str.title())
    return df


out = clean_naive(register)
print(f"{len(register):,} rows in, {len(out):,} rows out")

7,454 rows in, 5,793 rows out

Three lines, three different decisions, none of them written down anywhere:

• rows with no coordinates were deleted — more than a fifth of the register. Were they worth keeping?
• rows sharing a name and postcode were dropped, keeping the first — first in what order, and were they really the same business?
• every name was rewritten in title case — how many changed, and were any of them made wrong?

The last one is easy to check:

changed = register["name"] != register["name"].str.title()

print(f"{changed.sum():,} names rewritten, including:")

for pattern in ("KFC", "Sainsbury's", "McDonnell", " RC "):
    n = register.loc[changed & register["name"].str.contains(pattern, regex=False), "name"].iloc[0]
    print(f"  {n!r:42} → {n.title()!r}")

998 names rewritten, including:
  'KFC (Hunslet)'                            → 'Kfc (Hunslet)'
  "Sainsbury's"                              → "Sainsbury'S"
  'Mark McDonnell Family Butcher'            → 'Mark Mcdonnell Family Butcher'
  'Catering Leeds (St Theresas RC Primary)'  → 'Catering Leeds (St Theresas Rc Primary)'

The problem is not the operations. It is that "cleaning" quietly bundles three fundamentally different actions, and choosing between them per problem — rather than defaulting to whichever is shortest to write — is what separates a cleaning step you can defend from one you cannot.

Quick answer

Every cleaning rule does one of three things. Decide which, per problem, before writing it:

from dataclasses import dataclass, field


@dataclass
class CleanResult:
    data: gpd.GeoDataFrame
    rejects: pd.DataFrame = None
    repaired: dict = field(default_factory=dict)   # what changed, and how many
    flagged: dict = field(default_factory=dict)    # what is suspect, and how many
    rejected: dict = field(default_factory=dict)   # what was removed, and why


def clean(gdf) -> CleanResult:
    r = CleanResult(gdf.copy())

    # REJECT — no coordinates: nothing spatial can be done with the row, so it leaves the map
    no_geom = r.data.geometry.isna()
    r.rejected["no_coordinates"] = int(no_geom.sum())
    r.rejects = r.data[no_geom].assign(reason="no_coordinates")
    r.data = r.data[~no_geom]

    # REPAIR — a run of spaces inside a name is never meaningful
    squashed = r.data["name"].str.replace(r"\s+", " ", regex=True).str.strip()
    r.repaired["name_whitespace"] = int((squashed != r.data["name"]).sum())
    r.data["name"] = squashed

    # FLAG — same name at the same postcode: one business twice, or two kitchens?
    dupes = r.data.duplicated(subset=["name", "postcode"], keep=False)
    r.flagged["duplicate_name_postcode"] = int(dupes.sum())
    r.data["flag_duplicate_name_postcode"] = dupes
    return r


result = clean(register)
print("rejected:", result.rejected)
print("repaired:", result.repaired)
print("flagged: ", result.flagged)
assert len(result.data) == len(register) - sum(result.rejected.values())

rejected: {'no_coordinates': 1649}
repaired: {'name_whitespace': 53}
flagged:  {'duplicate_name_postcode': 22}

That assertion is the whole point: rows in = rows out + rows rejected, every rejection has a named reason, and the rejected rows still exist in result.rejects. The counts reconcile, so nobody has to wonder where sixteen hundred businesses went.

The three actions

Repair when determined, flag for a person, reject when unusable

Repair when the answer is determined, not chosen

A repair is legitimate when the correct output follows from a rule, and any competent person applying that rule would produce the same result.

# determined — whitespace inside or around a name is never meaningful
gdf["name"] = gdf["name"].str.replace(r"\s+", " ", regex=True).str.strip()

# determined — the CRS is declared, the target is declared
gdf = gdf.to_crs(27700)

# determined — the OGC rules say what a valid version of a self-intersecting polygon is
gdf["geometry"] = gdf.geometry.make_valid()

# NOT determined — is "KFC" an acronym? Is "McDonald's" one word?
gdf["name"] = gdf["name"].str.title()

That last line looks like a repair and is a guess, and the problem statement showed what it does to real names. The register also shows the subtler case — postcodes cut short:

pc = register["postcode"].fillna("")
full = pc.str.fullmatch(r"[A-Z]{1,2}\d[A-Z\d]? \d[A-Z]{2}")
print(f"{(~full & (pc != '')).sum():,} postcodes are not a full postcode, e.g. "
      f"{pc[~full & (pc != '')].head(4).tolist()}")
print(f"of those, by business type:")
print(register.loc[~full & (pc != ""), "type"].value_counts().head(3).to_string())

1,381 postcodes are not a full postcode, e.g. ['LS11', 'LS8 ', 'LS20', 'LS24']
of those, by business type:
type
Other catering premises    665
Retailers - other          335
Mobile caterer             288

Stripping the trailing space from 'LS9 ' is determined. Completing it is not: the full postcode is not in the data, and for most of these rows it is withheld on purpose — the FSA does not publish the full address of a business run from a private home, which is also why the same rows mostly have no coordinates. That is a flag, not a repair.

Two obligations come with every repair — count it, and make sure running it twice changes nothing the second time:

once = clean(register).data
twice = clean(once)
print(f"second pass repaired {twice.repaired}, rejected {twice.rejected}")

second pass repaired {'name_whitespace': 0}, rejected {'no_coordinates': 0}

Flag when a human has to decide

Flagging keeps the row and records the doubt. It is the right answer whenever the problem is a question about the world rather than a defect in the encoding.

data = result.data
data["flag_partial_postcode"] = ~data["postcode"].fillna("").str.fullmatch(r"[A-Z]{1,2}\d[A-Z\d]? \d[A-Z]{2}")
data["flag_not_yet_rated"] = ~data["rating"].str.fullmatch(r"\d")
data["flag_outside_leeds"] = ~data.geometry.within(leeds)
print(data.filter(like="flag_").sum().to_string())

flag_duplicate_name_postcode     22
flag_partial_postcode            48
flag_not_yet_rated              895
flag_outside_leeds                2

Flags are cheap to add and cost nothing to ignore, which is their weakness: a flag nobody consumes is a comment. Make them load-bearing:

# the analysis states which flags it tolerates
usable = data[~data["flag_outside_leeds"] & ~data["flag_not_yet_rated"]]
print(f"analysis of ratings uses {len(usable):,} of {len(data):,} mapped businesses")

analysis of ratings uses 4,908 of 5,805 mapped businesses

Now the exclusion is visible at the point of use, in the code that depends on it — rather than buried in a cleaning function three modules away. A map of every business would tolerate flag_not_yet_rated; an analysis of hygiene ratings cannot.

A useful convention: one boolean column per problem, prefixed flag_, never a single free-text notes column. Booleans can be counted, filtered and asserted on; prose cannot.

Reject only when the row cannot be used at all

Rejection is the most destructive option and the most commonly over-used, because deleting a row makes the error message go away. The register's rows without coordinates are a fair rejection from a map, and it is worth seeing what they are before deciding:

print(result.rejects["type"].value_counts().head(4).to_string())
result.rejects.drop(columns="geometry").to_csv("rejected_rows.csv", index=False)

type
Other catering premises    681
Retailers - other          380
Mobile caterer             297
Restaurant/Cafe/Canteen    112

Mobile caterers have no fixed address to plot, and much of the rest is catering at private or undisclosed premises. They are perfectly good rows for a count of businesses by type — so the rejection belongs to the map, not to the register, and the rejects file keeps them for everything else.

Illegitimate rejections — these are flags wearing a rejection's clothes:

gdf = gdf[gdf.is_valid]                                   # deletes real features rather than repairing them
gdf = gdf.drop_duplicates(["name", "postcode"])           # picks a winner by file order

gdf = gdf[gdf["rating"].str.fullmatch(r"\d")]             # deletes every business awaiting inspection or exempt
gdf = gdf[gdf["postcode"].str.len() > 4]                  # deletes home businesses for having privacy

Three rules make rejection defensible:

1. Count it, by reason. {"no_coordinates": 1649}, never a single total.
2. Keep the rows somewhere. A rejects file costs nothing and answers every later question.
3. Never reject silently in a loop. That is quarantine at the row level, and it needs the same discipline.

The order matters, and it is not obvious

Each action changes what the next rule sees, so the sequence is part of the design:

# 1. reject what cannot participate at all
#    (no geometry, empty geometry — nothing downstream can use them)
# 2. repair what is determined
#    (validity, CRS, whitespace — makes the flags below meaningful)
# 3. flag what needs judgement
#    (duplicates, boundary checks — computed on repaired data, so they are real)
# 4. reject again if repair produced something unusable
#    (make_valid can return an empty geometry)

The register shows why on its first rule. Run the boundary flag before rejecting the rows without coordinates:

too_early = ~register.geometry.within(leeds)
print(f"flag_outside_leeds before rejecting: {too_early.sum():,} rows; after: {data['flag_outside_leeds'].sum()}")

flag_outside_leeds before rejecting: 1,651 rows; after: 2

A missing point is not within anything, so every business without coordinates is "outside Leeds" — and the flag that should find two geocoding errors finds sixteen hundred. Flagging duplicates before squashing whitespace has the opposite effect: two spellings of the same name stay apart.

Never let cleaning be silent

def summarise(result: CleanResult) -> str:
    lines = [f"{len(result.data):,} rows out"]
    for kind, counts in [("rejected", result.rejected), ("repaired", result.repaired),
                         ("flagged", result.flagged)]:
        for reason, n in counts.items():
            if n:
                lines.append(f"  {kind:9s} {n:>6,}  {reason}")
    return "\n".join(lines)


print(summarise(result))

5,805 rows out
  rejected   1,649  no_coordinates
  repaired      53  name_whitespace
  flagged       22  duplicate_name_postcode

A handful of lines that make the step auditable. Without them, the only visible fact is that a number changed.

Rules as data on the Leeds food hygiene register

The data used on this page

The Food Standards Agency's Food Hygiene Rating Scheme register for Leeds City Council, fetched from the FSA's public API: every inspected food business, with its name, type, postcode, rating and — where published — coordinates, which are projected to British National Grid. Rows without coordinates keep a null geometry. The Leeds district boundary (ONS wards, December 2024) is used to check the points.

import geopandas as gpd
import pandas as pd
import requests
from shapely import Point

ONS = "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services"
leeds = gpd.read_file(f"{ONS}/Wards_December_2024_Boundaries_UK_BFC/FeatureServer/0/query"
                      "?where=LAD24NM%3D%27Leeds%27&outFields=WD24CD&outSR=27700&f=geojson")
leeds = leeds.set_crs(27700, allow_override=True).union_all()

rows, page = [], 1
while True:                                          # the API returns at most 5,000 per page
    batch = requests.get("https://api.ratings.food.gov.uk/Establishments", headers={"x-api-version": "2"},
                         params={"localAuthorityId": 397, "pageSize": 5000, "pageNumber": page},
                         timeout=120).json()["establishments"]
    rows += batch
    if len(batch) < 5000:
        break
    page += 1

register = gpd.GeoDataFrame(
    {"fhrsid": [e["FHRSID"] for e in rows],
     "name": [e["BusinessName"] for e in rows],
     "type": [e["BusinessType"] for e in rows],
     "postcode": [e["PostCode"] for e in rows],
     "rating": [e["RatingValue"] for e in rows]},
    geometry=[Point(float(e["geocode"]["longitude"]), float(e["geocode"]["latitude"]))
              if e["geocode"]["latitude"] else None for e in rows],
    crs=4326).to_crs(27700).sort_values("fhrsid").reset_index(drop=True)
print(f"{len(register):,} businesses, {register.geometry.isna().sum():,} without coordinates")

7,454 businesses, 1,649 without coordinates

Example 1: rules as data, not as code

Once each rule declares its action, the cleaning function stops being a wall of statements:

from typing import Callable, Literal

POSTCODE = r"[A-Z]{1,2}\d[A-Z\d]? \d[A-Z]{2}"


@dataclass
class Rule:
    name: str
    action: Literal["repair", "flag", "reject"]
    test: Callable                    # rows where this is True have the problem
    column: str | None = None         # what a repair rewrites
    fix: Callable | None = None       # required for repair


RULES = [
    Rule("no_coordinates", "reject", lambda g: g.geometry.isna()),
    Rule("name_whitespace", "repair",
         lambda g: g["name"].str.replace(r"\s+", " ", regex=True).str.strip() != g["name"],
         "name", lambda g: g["name"].str.replace(r"\s+", " ", regex=True).str.strip()),
    Rule("postcode_whitespace", "repair",
         lambda g: g["postcode"].fillna("").str.strip() != g["postcode"].fillna(""),
         "postcode", lambda g: g["postcode"].str.strip()),
    Rule("duplicate_name_postcode", "flag", lambda g: g.duplicated(["name", "postcode"], keep=False)),
    Rule("partial_postcode", "flag", lambda g: ~g["postcode"].fillna("").str.fullmatch(POSTCODE)),
    Rule("not_yet_rated", "flag", lambda g: ~g["rating"].str.fullmatch(r"\d")),
    Rule("outside_leeds", "flag", lambda g: ~g.geometry.within(leeds)),
]


def apply_rules(gdf, rules=RULES) -> CleanResult:
    r = CleanResult(gdf.copy(), rejects=gdf.iloc[:0].assign(reason=""))
    for rule in rules:
        hit = rule.test(r.data)
        n = int(hit.sum())
        if rule.action == "reject":
            r.rejected[rule.name] = n
            r.rejects = pd.concat([r.rejects, r.data[hit].assign(reason=rule.name)])
            r.data = r.data[~hit]
        elif rule.action == "repair":
            r.repaired[rule.name] = n
            r.data.loc[hit, rule.column] = rule.fix(r.data.loc[hit])
        else:
            r.flagged[rule.name] = n
            r.data[f"flag_{rule.name}"] = hit
    return r


result = apply_rules(register)
print(summarise(result))

5,805 rows out
  rejected   1,649  no_coordinates
  repaired      53  name_whitespace
  flagged       22  duplicate_name_postcode
  flagged       48  partial_postcode
  flagged      895  not_yet_rated
  flagged        2  outside_leeds

Now the rules are reviewable in one place, a new rule is one entry, and the action is declared rather than implied by which statement someone reached for. It is also the shape a config file can hold. The column field is there because an earlier version of this code derived the column from the rule's name, and every rule name that did not end in a column name repaired the wrong column.

Example 2: proving the counts reconcile

def assert_reconciles(before, after, rejected):
    assert len(after) + rejected == len(before), (
        f"{len(before):,} in, {len(after):,} out, {rejected:,} rejected — "
        f"{len(before) - len(after) - rejected:,} rows unaccounted for"
    )


assert_reconciles(register, result.data, sum(result.rejected.values()))
print("rules engine: reconciles")
try:
    assert_reconciles(register, clean_naive(register), register.geometry.isna().sum())
except AssertionError as err:
    print(f"naive clean:  {err}")

rules engine: reconciles
naive clean:  7,454 in, 5,793 out, 1,649 rejected — 12 rows unaccounted for

Rows disappearing without a recorded reason is the single most common defect in cleaning code, and one assertion catches all of it. The naive function from the problem statement fails it: even crediting it with the rows it dropped for having no coordinates, its drop_duplicates removed more that nobody recorded. Run the check in the pipeline, not just in tests.

Example 3: idempotence, which repairs must have and rejections give free

def test_cleaning_is_idempotent(register):
    once = apply_rules(register).data
    twice = apply_rules(once)

    assert len(twice.data) == len(once)          # nothing left to reject
    assert not any(twice.repaired.values())      # nothing left to repair
    assert not any(twice.rejected.values())


test_cleaning_is_idempotent(register)
print("idempotent: a second pass rejects and repairs nothing")

idempotent: a second pass rejects and repairs nothing

If the second pass still repairs something, a rule is not converging — usually one repair producing a value that another rule then alters. That is worth knowing before the job runs nightly and rewrites the same rows forever. See idempotency explained.

What each action does to the evidence

The three actions differ in what they do to the evidence, and that is the useful way to think about them.

A repair changes the record and keeps the row. The original value is gone unless you saved it, so a repair is only safe where the original was unambiguously wrong — a self-intersecting ring, a name with trailing whitespace, a CRS label that contradicts the coordinates. In every one of those, no information is lost because the original carried none.

A rejection removes the row and, with it, the evidence that the row existed. This is why an unrecorded rejection is so corrosive: the dataset now says something false by omission, and there is no way to detect it from the output alone. A rejected row that is counted and written to a rejects file loses nothing; a rejected row that is filtered away in a comprehension is unrecoverable.

A flag changes nothing and adds information. It is the only one of the three that is reversible, which makes it the right default whenever you are unsure — and the reason "when in doubt, flag" is better advice than "when in doubt, drop".

The deeper reason to make the choice explicit is that cleaning encodes a judgement about the data's purpose. Dropping parcels smaller than a square metre is correct for a land-use area calculation and wrong for a completeness audit. Filling a missing ward name with "UNKNOWN" is correct for a groupby that must total correctly and wrong for a mailing list. The same input, the same defect, two opposite right answers.

A cleaning function that does not state which action it took cannot be reviewed against a purpose, because the purpose was never written down. Naming the action per rule forces it into the open — which is most of the value, before a single count is printed.

Area-changing repairs, unsorted dedupes and broken keys

• A repair that changes area or row count is not really a repair. make_valid on a bow-tie changes its area and can split it into a MultiPolygon; measure and report it.
• drop_duplicates without a sort is non-deterministic. Sort first, or the same input gives different output across runs.
• Flags need a naming convention. flag_ prefixed booleans are countable; a free-text notes column is not.
• Rejecting a row can break a foreign key. If another table references it, rejection is a cascade, not a filter.
• "Reject" and "quarantine" are the same idea at different scales — rows within a file, files within a batch. See failure policy in batch processing.
• Repairs must converge. A rule that repairs the output of another rule forever will rewrite the dataset on every run.
• Keep the rejects file next to the output, not in a temp directory. It is part of the result.
• Do not clean in the reader. A read_and_clean() function makes it impossible to see what arrived versus what you changed.

Internal links

• The Python GIS data cleaning checklist — the rules this page tells you how to classify
• How to build a repeatable data-cleaning report in GeoPandas — publishing the counts
• Null, empty, missing and invalid: four kinds of broken geometry — the states each rule tests for
• How to validate a GeoDataFrame against a schema before analysis — asserting instead of cleaning
• Failure policy in batch processing — the same three choices, one level up
• Idempotency explained: why a GIS job must be safe to re-run — why repairs must converge
• Spatial data quality: the six dimensions that matter — what you are cleaning towards
• Configuration vs code: what belongs in a config file — where a rules table should live

FAQ

What is the default when I cannot decide?

Flag. It is the only reversible option: the row survives, the problem is recorded, and a later decision can still go either way.

How do I stop rows vanishing without explanation?

Assert that rows in equals rows out plus rejections, and record rejections by reason. One assertion catches every unrecorded deletion.

Should the cleaning function return counts or log them?

Return them. A returned object can be asserted on in tests, written to a run log, and compared with yesterday's; a printed line can only be read.

What is the difference between a flag and a rejection?

A flag keeps the row and records doubt, leaving the decision to the analysis. A rejection makes the decision on the analysis's behalf, permanently.

How many flags is too many?

If most rows carry a flag, the flag is describing the dataset rather than an exception — which usually means the rule is wrong, or the data needs a conversation rather than a filter.

Where should rejected rows go?

A CSV or GeoPackage beside the output, with the reason column included. It costs nothing and answers "what happened to parcel 4102" instantly.