Repair, Reject or Flag? Choosing What Cleaning Should Do

Problem statement

The cleaning function works. Nobody can say what it did.

def clean(gdf):
    gdf = gdf[gdf.geometry.notna()]
    gdf["geometry"] = gdf.geometry.make_valid()
    gdf = gdf.drop_duplicates(subset="parcel_id")
    gdf["ward"] = gdf["ward"].str.strip().str.title()
    return gdf

Four lines, four different decisions, none of them written down anywhere:

rows with no geometry were deleted — was that right, or did they have attributes worth keeping?
invalid geometries were repaired — did any of them change area?
duplicate ids were dropped, keeping the first — first in what order?
ward names were rewritten — how many changed, and did any two now collide?

Downstream, 12,400 rows became 12,180. Nobody knows which 220 or why. And the function is idempotent only by luck.

The problem is not the operations. It is that "cleaning" quietly bundles three fundamentally different actions, and choosing between them per problem — rather than defaulting to whichever is shortest to write — is what separates a cleaning step you can defend from one you cannot.

Quick answer

Every cleaning rule does one of three things. Decide which, per problem, before writing it:

Action	Do it when	Cost
Repair — change the data to something correct	the correct value is derivable without judgement	you have changed the data; say so
Flag — keep the row, mark the problem	the row is usable but suspect, or a human must decide	downstream must honour the flag
Reject — remove the row from this dataset	the row cannot be used and cannot be fixed	you have deleted evidence; count it

from dataclasses import dataclass, field

@dataclass
class CleanResult:
    data: "gpd.GeoDataFrame"
    repaired: dict = field(default_factory=dict)   # what changed, and how many
    flagged: dict = field(default_factory=dict)    # what is suspect, and how many
    rejected: dict = field(default_factory=dict)   # what was removed, and why

def clean(gdf) -> CleanResult:
    r = CleanResult(gdf.copy())

    # REJECT — no geometry, and no way to derive one
    no_geom = r.data.geometry.isna()
    r.rejected["null_geometry"] = int(no_geom.sum())
    r.data = r.data[~no_geom]

    # REPAIR — the OGC rules define the correct answer
    invalid = ~r.data.is_valid
    r.repaired["invalid_geometry"] = int(invalid.sum())
    r.data.loc[invalid, "geometry"] = r.data.loc[invalid, "geometry"].make_valid()

    # FLAG — a duplicate id is a records problem, not a geometry one
    dupes = r.data["parcel_id"].duplicated(keep=False)
    r.flagged["duplicate_id"] = int(dupes.sum())
    r.data["flag_duplicate_id"] = dupes

    return r

result = clean(parcels)
print(result.rejected)   # {'null_geometry': 6}
print(result.repaired)   # {'invalid_geometry': 31}
print(result.flagged)    # {'duplicate_id': 44}
assert len(result.data) == len(parcels) - sum(result.rejected.values())

That assertion is the whole point: rows in = rows out + rows rejected, and every rejection has a named reason. The counts reconcile, so nobody has to wonder where 220 rows went.

The three actions

Reject deletes evidence. Repair changes the record. Flag does neither, and defers the decision.

Step-by-step solution

The question that decides it: is the correct value derivable without judgement?

Repair when the answer is determined, not chosen

A repair is legitimate when the correct output follows from a rule, and any competent person applying that rule would produce the same result.

# determined — the OGC rules say what a valid version of this shape is
gdf["geometry"] = gdf.geometry.make_valid()

# determined — the CRS is declared, the target is declared
gdf = gdf.to_crs(27700)

# determined — whitespace around a name is never meaningful
gdf["ward"] = gdf["ward"].str.strip()

# NOT determined — which of two conflicting areas is right?
gdf["area_m2"] = gdf["area_m2"].fillna(gdf.geometry.area)   # is the stored one wrong, or is the geometry?

That last line looks like a repair and is a guess. The stored area_m2 might be authoritative and the geometry a rough sketch. Filling from geometry silently asserts otherwise.

Two obligations come with every repair:

# 1. count it
before = gdf.geometry.area.sum()
gdf["geometry"] = gdf.geometry.make_valid()
after = gdf.geometry.area.sum()
report["repaired"] = {"invalid": int(invalid.sum()),
                      "area_change_m2": round(after - before, 2)}

# 2. make it idempotent — running twice must change nothing the second time
assert gdf.is_valid.all()

The area delta is the check people skip. make_valid on a bowtie polygon can change its area substantially, and a repair that quietly moves 4,000 m² is a repair somebody should have been told about.

Flag when a human has to decide

Flagging keeps the row and records the doubt. It is the right answer whenever the problem is a disagreement about the world rather than a defect in the encoding.

gdf["flag_overlap"] = gdf.index.isin(overlapping_ids)
gdf["flag_area_mismatch"] = (gdf["area_m2"] - gdf.geometry.area).abs() > 100
gdf["flag_outside_boundary"] = ~gdf.geometry.within(study_area)
gdf["flag_no_name"] = gdf["ward"].isna()

Flags are cheap to add and cost nothing to ignore, which is their weakness: a flag nobody consumes is a comment. Make them load-bearing:

# the analysis states which flags it tolerates
usable = gdf[~gdf["flag_overlap"] & ~gdf["flag_outside_boundary"]]
print(f"analysis uses {len(usable):,} of {len(gdf):,} features")

Now the exclusion is visible at the point of use, in the code that depends on it — rather than buried in a cleaning function three modules away.

A useful convention: one boolean column per problem, prefixed flag_, never a single free-text notes column. Booleans can be counted, filtered and asserted on; prose cannot.

Reject only when the row cannot be used at all

Rejection is the most destructive option and the most commonly over-used, because deleting a row makes the error message go away.

Legitimate rejections:

gdf = gdf[gdf.geometry.notna()]        # no geometry: nothing spatial can be done
gdf = gdf[~gdf.geometry.is_empty]      # empty geometry: matches nothing, contributes nothing
gdf = gdf[gdf["parcel_id"].notna()]    # no key: cannot be joined or traced

Illegitimate ones — these are flags wearing a rejection's clothes:

gdf = gdf[gdf.is_valid]                     # deletes real parcels rather than repairing them
gdf = gdf.drop_duplicates("parcel_id")      # picks a winner by file order
gdf = gdf[gdf.geometry.area > 1]            # deletes small parcels; some are real
gdf = gdf[gdf["ward"].notna()]              # deletes rows whose only fault is a missing label

Three rules make rejection defensible:

Count it, by reason. rejected = {"null_geometry": 6, "no_key": 2}, never a single total.
Keep the rows somewhere. A rejects file costs nothing and answers every later question.
Never reject silently in a loop. That is quarantine at the row level, and it needs the same discipline.

rejects = gdf[no_geom]
rejects.drop(columns="geometry").to_csv(out / "rejected_rows.csv", index=False)

The order matters, and it is not obvious

Each action changes what the next rule sees, so the sequence is part of the design:

# 1. reject what cannot participate at all
#    (nulls, empties — nothing downstream can use them)
# 2. repair what is determined
#    (validity, CRS, whitespace — makes the flags below meaningful)
# 3. flag what needs judgement
#    (overlaps, mismatches — computed on repaired geometry, so they are real)
# 4. reject again if repair produced something unusable
#    (make_valid can return an empty geometry)

Flagging overlaps before repairing validity produces false positives, because an invalid self-intersecting polygon overlaps things it does not really overlap. Repairing before rejecting empties means repairing rows about to be deleted. The order above avoids both.

Never let cleaning be silent

def summarise(result: CleanResult) -> str:
    lines = [f"{len(result.data):,} rows out"]
    for kind, counts in [("rejected", result.rejected),
                         ("repaired", result.repaired),
                         ("flagged", result.flagged)]:
        for reason, n in counts.items():
            if n:
                lines.append(f"  {kind:9s} {n:>6,}  {reason}")
    return "\n".join(lines)

12,180 rows out
  rejected       6  null_geometry
  rejected     214  empty_geometry
  repaired      31  invalid_geometry
  repaired    4,102  whitespace_in_ward
  flagged       44  duplicate_id
  flagged      214  overlapping

Six lines that make the step auditable. Without them, the only visible fact is that a number changed.

Code examples

Example 1: rules as data, not as code

Once each rule declares its action, the cleaning function stops being a wall of statements:

from dataclasses import dataclass
from typing import Callable, Literal

@dataclass
class Rule:
    name: str
    action: Literal["repair", "flag", "reject"]
    test: Callable          # rows where this is True have the problem
    fix: Callable | None = None    # required for repair

RULES = [
    Rule("null_geometry",  "reject", lambda g: g.geometry.isna()),
    Rule("empty_geometry", "reject", lambda g: g.geometry.is_empty),
    Rule("invalid_geometry", "repair",
         lambda g: ~g.is_valid,
         lambda g: g.geometry.make_valid()),
    Rule("untrimmed_ward", "repair",
         lambda g: g["ward"].fillna("").str.strip().ne(g["ward"].fillna("")),
         lambda g: g["ward"].str.strip()),
    Rule("duplicate_id", "flag", lambda g: g["parcel_id"].duplicated(keep=False)),
    Rule("area_mismatch", "flag",
         lambda g: (g["area_m2"] - g.geometry.area).abs() > 100),
]

def apply_rules(gdf, rules=RULES) -> CleanResult:
    r = CleanResult(gdf.copy())
    for rule in rules:
        hit = rule.test(r.data)
        n = int(hit.sum())
        if not n:
            continue
        if rule.action == "reject":
            r.rejected[rule.name] = n
            r.data = r.data[~hit]
        elif rule.action == "repair":
            r.repaired[rule.name] = n
            target = "geometry" if "geometry" in rule.name else rule.name.split("_")[-1]
            r.data.loc[hit, target] = rule.fix(r.data.loc[hit])
        else:
            r.flagged[rule.name] = n
            r.data[f"flag_{rule.name}"] = hit
    return r

Now the rules are reviewable in one place, a new rule is one entry, and the action is declared rather than implied by which statement someone reached for. It is also the shape a config file can hold.

Example 2: proving the counts reconcile

def assert_reconciles(before, result: CleanResult):
    rejected = sum(result.rejected.values())
    assert len(result.data) + rejected == len(before), (
        f"{len(before):,} in, {len(result.data):,} out, {rejected:,} rejected — "
        f"{len(before) - len(result.data) - rejected:,} rows unaccounted for"
    )

Rows disappearing without a recorded reason is the single most common defect in cleaning code, and one assertion catches all of it. Run it in the pipeline, not just in tests.

Example 3: idempotence, which repairs must have and rejections give free

def test_cleaning_is_idempotent(dirty_parcels):
    once = apply_rules(dirty_parcels).data
    twice = apply_rules(once)

    assert len(twice.data) == len(once)          # nothing left to reject
    assert not any(twice.repaired.values())      # nothing left to repair
    assert not any(twice.rejected.values())

If the second pass still repairs something, a rule is not converging — usually make_valid producing a shape that another rule then alters. That is worth knowing before the job runs nightly and rewrites the same rows forever. See idempotency explained.

Explanation

Checklist of the questions an unreported cleaning step cannot answer. — Every line is a question somebody asks eventually, usually under time pressure.

The three actions differ in what they do to the evidence, and that is the useful way to think about them.

A repair changes the record and keeps the row. The original value is gone unless you saved it, so a repair is only safe where the original was unambiguously wrong — a self-intersecting ring, a name with trailing whitespace, a CRS label that contradicts the coordinates. In every one of those, no information is lost because the original carried none.

A rejection removes the row and, with it, the evidence that the row existed. This is why an unrecorded rejection is so corrosive: the dataset now says something false by omission, and there is no way to detect it from the output alone. A rejected row that is counted and written to a rejects file loses nothing; a rejected row that is filtered away in a comprehension is unrecoverable.

A flag changes nothing and adds information. It is the only one of the three that is reversible, which makes it the right default whenever you are unsure — and the reason "when in doubt, flag" is better advice than "when in doubt, drop".

The deeper reason to make the choice explicit is that cleaning encodes a judgement about the data's purpose. Dropping parcels smaller than a square metre is correct for a land-use area calculation and wrong for a completeness audit. Filling a missing ward name with "UNKNOWN" is correct for a groupby that must total correctly and wrong for a mailing list. The same input, the same defect, two opposite right answers.

A cleaning function that does not state which action it took cannot be reviewed against a purpose, because the purpose was never written down. Naming the action per rule forces it into the open — which is most of the value, before a single count is printed.

Edge cases or notes

A repair that changes area or row count is not really a repair. make_valid on a bowtie changes both; measure and report it.
drop_duplicates without a sort is non-deterministic. Sort first, or the same input gives different output across runs.
Flags need a naming convention. flag_ prefixed booleans are countable; a free-text notes column is not.
Rejecting a row can break a foreign key. If another table references it, rejection is a cascade, not a filter.
"Reject" and "quarantine" are the same idea at different scales — rows within a file, files within a batch. See failure policy in batch processing.
Repairs must converge. A rule that repairs the output of another rule forever will rewrite the dataset on every run.
Keep the rejects file next to the output, not in a temp directory. It is part of the result.
Do not clean in the reader. A read_and_clean() function makes it impossible to see what arrived versus what you changed.

Internal links

The Python GIS data cleaning checklist — the rules this page tells you how to classify
How to build a repeatable data-cleaning report in GeoPandas — publishing the counts
Null, empty, missing and invalid: four kinds of broken geometry — the states each rule tests for
How to validate a GeoDataFrame against a schema before analysis — asserting instead of cleaning
Failure policy in batch processing — the same three choices, one level up
Idempotency explained: why a GIS job must be safe to re-run — why repairs must converge
Spatial data quality: the six dimensions that matter — what you are cleaning towards
Configuration vs code: what belongs in a config file — where a rules table should live

FAQ

What is the default when I cannot decide?

Flag. It is the only reversible option: the row survives, the problem is recorded, and a later decision can still go either way.

Is dropping invalid geometries ever right?

Rarely. Invalid geometry is usually a digitising artefact that make_valid fixes exactly. Dropping deletes a real feature to avoid an error message.

How do I stop rows vanishing without explanation?

Assert that rows in equals rows out plus rejections, and record rejections by reason. One assertion catches every unrecorded deletion.

Should the cleaning function return counts or log them?

Return them. A returned object can be asserted on in tests, written to a run log, and compared with yesterday's; a printed line can only be read.

What is the difference between a flag and a rejection?

A flag keeps the row and records doubt, leaving the decision to the analysis. A rejection makes the decision on the analysis's behalf, permanently.

How many flags is too many?

If most rows carry a flag, the flag is describing the dataset rather than an exception — which usually means the rule is wrong, or the data needs a conversation rather than a filter.

Where should rejected rows go?

A CSV or GeoPackage beside the output, with the reason column included. It costs nothing and answers "what happened to parcel 4102" instantly.

Repair, Reject or Flag? Choosing What Cleaning Should Do #

Problem statement #

Quick answer #

The three actions #

Step-by-step solution #

Repair when the answer is determined, not chosen #

Flag when a human has to decide #

Reject only when the row cannot be used at all #

The order matters, and it is not obvious #

Never let cleaning be silent #

Code examples #

Example 1: rules as data, not as code #

Example 2: proving the counts reconcile #

Example 3: idempotence, which repairs must have and rejections give free #

Explanation #

Edge cases or notes #

Internal links #

FAQ #

What is the default when I cannot decide? #

Is dropping invalid geometries ever right? #

How do I stop rows vanishing without explanation? #

Should the cleaning function return counts or log them? #

What is the difference between a flag and a rejection? #

How many flags is too many? #

Where should rejected rows go? #