🔧 Fix Data cleaning Geometry & topology Updated

My Cleaned Layer Has More Rows Than the Original: How to Fix It

Problem statement

Cleaning is supposed to remove things. The row count went up.

len(parcels)                    # 12,400
clean = clean_parcels(parcels)
len(clean)                      # 12,847

Four hundred and forty-seven rows appeared. Every total computed from the layer is now inflated, every parcel id appears more than once for some parcels, and the spatial join downstream produces duplicate rows that nobody ordered.

clean["parcel_id"].is_unique     # False
clean.groupby("ward").size().sum() - parcels.groupby("ward").size().sum()   # +447

The cleaning code contains no concat, no join and no loop that appends. It still multiplied rows, because three ordinary cleaning operations split one feature into several, and none of them warns.

Quick answer

Find which step changed the count, then decide whether the split was correct:

def trace_counts(gdf, steps):
    """Run each step in turn and report what it did to the row count."""
    n = len(gdf)
    print(f"{'start':32s} {n:>7,}")
    for name, fn in steps:
        gdf = fn(gdf)
        delta = len(gdf) - n
        flag = "  ←" if delta else ""
        print(f"{name:32s} {len(gdf):>7,}  {delta:+,}{flag}")
        n = len(gdf)
    return gdf

trace_counts(parcels, [
    ("drop null geometry",   lambda g: g[g.geometry.notna()]),
    ("make_valid",           lambda g: g.assign(geometry=g.geometry.make_valid())),
    ("explode",              lambda g: g.explode(index_parts=False)),
    ("drop empty",           lambda g: g[~g.geometry.is_empty]),
])
start                             12,400
drop null geometry                12,394  -6
make_valid                        12,394  +0
explode                           12,847  +453  ←
drop empty                        12,847  +0
Operation Why it adds rows Fix
explode() one MultiPolygon → one row per part add a stable id before, dissolve after, or do not explode
overlay() / intersection() one input feature → one row per output piece aggregate back by the original id
sjoin() one row per matching pair deduplicate, or aggregate the right side
make_valid() a bowtie becomes a MultiPolygon (row count same, parts change) check geom_type after
dissolve().explode() the classic round-trip that changes counts twice keep the dissolve key 
# the usual culprit, and the usual fix
parcels["feature_id"] = parcels["parcel_id"]        # stable id BEFORE exploding
parts = parcels.explode(index_parts=False)
back = parts.dissolve(by="feature_id", aggfunc="first").reset_index()
assert len(back) == len(parcels)

Where the extra rows come from

Grid of operations that change row counts, showing what one input row becomes.
None of these are bugs. All of them are surprises if you did not expect the split.

Step-by-step solution

Triage rows pairing each row-count symptom with its cause and its fix.
The count tells you which operation; the id tells you whether it was correct.

1. Find the step, not the cause

Do not read the code looking for the bug. Instrument it — the count is the fastest diagnostic available, and it points at exactly one line.

def counted(fn, label):
    """Wrap any step so it reports what it did to the row count."""
    def wrapper(gdf, *a, **kw):
        before = len(gdf)
        out = fn(gdf, *a, **kw)
        if len(out) != before:
            print(f"  {label}: {before:,} → {len(out):,} ({len(out)-before:+,})")
        return out
    return wrapper

Applied once, this turns "somewhere in cleaning" into "line 34".

2. explode() — the most common cause by far 

print(parcels.geom_type.value_counts())
# Polygon         12103
# MultiPolygon      297      ← 297 features holding 750 parts

parts = parcels.explode(index_parts=False)
len(parts)                    # 12,853

explode is doing exactly what it says: one row per single-part geometry. The question is whether you wanted that.

You want it when the parts are independent things — separate buildings recorded as one multi-feature, islands that should be counted individually.

You do not want it when the parts are one thing — a parcel split by a road, a borough with an offshore island. Those must stay one row or every count and every sum is wrong.

If you exploded for a reason and need to get back:

parcels = parcels.reset_index(drop=True)
parcels["feature_id"] = parcels.index               # stable, before exploding

parts = parcels.explode(index_parts=False)
parts["part_area"] = parts.geometry.area            # do the per-part work

# ... filter slivers, classify parts, whatever needed the split ...
parts = parts[parts["part_area"] > 1.0]

back = parts.dissolve(by="feature_id", aggfunc="first").reset_index()
print(f"{len(parcels)} → {len(parts)} parts → {len(back)} features")

Without feature_id, the round trip is impossible — after exploding, nothing records which parts belonged together. See how to split multipart geometries.

3. overlay and intersection — one row per resulting piece 

clipped = gpd.overlay(parcels, wards, how="intersection")
len(clipped)                  # 13,102 — a parcel spanning two wards became two rows

That is correct behaviour: a parcel in two wards genuinely has two pieces, one in each. It becomes a bug when the next step sums area_m2 — a column carried over unchanged from before the split, so it is now counted twice.

# the trap
clipped["area_m2"].sum()              # inflated — the stored area came along whole

# the fix: recompute anything derived from geometry, after the split
clipped["area_m2"] = clipped.geometry.area
clipped["area_m2"].sum()              # correct

Any column derived from geometry must be recomputed after any operation that changes geometry. Stored SHAPE_Area, length_m, perimeter — all of them are stale the moment a shape is clipped.

4. sjoin — one row per matching pair 

joined = gpd.sjoin(parcels, wards, predicate="within")
len(joined)                   # 12,444 — 44 parcels matched two wards

A parcel exactly on a ward boundary is within both, so it produces two rows. See spatial predicates explained for why, and duplicate rows from sjoin for the full treatment.

# detect
dupes = joined.index.duplicated(keep=False)
print(f"{dupes.sum()} rows from {joined.index[dupes].nunique()} multi-matched parcels")

# resolve deterministically — never by file order
joined = (joined.sort_values(["parcel_id", "index_right"])
                .loc[lambda d: ~d.index.duplicated(keep="first")])

Or aggregate instead of choosing, when both matches are real:

wards_per_parcel = (joined.groupby(joined.index)["name"]
                          .agg(lambda s: ";".join(sorted(set(s)))))

5. make_valid — the count stays, the parts change 

before_types = parcels.geom_type.value_counts()
parcels = parcels.assign(geometry=parcels.geometry.make_valid())
after_types = parcels.geom_type.value_counts()
print(pd.concat([before_types, after_types], axis=1, keys=["before", "after"]))
              before  after
Polygon        12103  12081
MultiPolygon     297    316      ← 22 bowties became multi-part
GeometryCollection   0      3      ← and 3 became collections

make_valid does not add rows, but it turns single-part polygons into multi-part ones — so a later explode produces more rows than it would have before. That is why a pipeline that ran fine for months suddenly multiplies rows after a supplier sends slightly worse geometry.

# keep only polygonal parts of any GeometryCollection
from shapely.geometry import GeometryCollection, MultiPolygon
def polygons_only(geom):
    if isinstance(geom, GeometryCollection):
        parts = [g for g in geom.geoms if g.geom_type in ("Polygon", "MultiPolygon")]
        return MultiPolygon([p for g in parts for p in getattr(g, "geoms", [g])]) if parts else None
    return geom

parcels["geometry"] = parcels.geometry.apply(polygons_only)

6. Assert the count you expect, at every boundary 

def assert_row_count(gdf, expected, label=""):
    assert len(gdf) == expected, (
        f"{label}: expected {expected:,} rows, got {len(gdf):,} ({len(gdf)-expected:+,})"
    )

def assert_key_unique(gdf, key="parcel_id"):
    dupes = gdf[key].duplicated()
    assert not dupes.any(), (
        f"{dupes.sum():,} duplicate {key} values, e.g. "
        f"{gdf.loc[dupes, key].head(3).tolist()}"
    )

assert_key_unique is the more valuable of the two. Row counts legitimately change; a primary key going non-unique never does, and it is the condition that actually breaks the downstream join.

Code examples

Example 1: a cleaning function that cannot silently multiply 

from dataclasses import dataclass, field

@dataclass
class CleanReport:
    counts: list = field(default_factory=list)

    def step(self, label, before, after):
        self.counts.append({"step": label, "before": before, "after": after,
                            "delta": after - before})

    def __str__(self):
        lines = [f"{'step':30s} {'before':>8s} {'after':>8s} {'delta':>7s}"]
        for c in self.counts:
            mark = " ←" if c["delta"] else ""
            lines.append(f"{c['step']:30s} {c['before']:>8,} {c['after']:>8,} "
                         f"{c['delta']:>+7,}{mark}")
        return "\n".join(lines)

def clean_parcels(gdf, *, key="parcel_id"):
    report = CleanReport()
    def track(label, out, before):
        report.step(label, before, len(out))
        return out

    n = len(gdf)
    gdf = track("drop null geometry", gdf[gdf.geometry.notna()], n)
    gdf = track("drop empty geometry", gdf[~gdf.geometry.is_empty], len(gdf))
    gdf = track("make_valid", gdf.assign(geometry=gdf.geometry.make_valid()), len(gdf))
    gdf = track("polygons only",
                gdf[gdf.geom_type.isin(["Polygon", "MultiPolygon"])], len(gdf))

    assert gdf[key].is_unique, f"{key} is no longer unique after cleaning"
    return gdf, report
clean, report = clean_parcels(parcels)
print(report)
step                             before    after   delta
drop null geometry               12,400   12,394      -6
drop empty geometry              12,394   12,180    -214
make_valid                       12,180   12,180      +0
polygons only                    12,180   12,177      -3

Every change accounted for, and the key assertion means a multiplication cannot get past this function.

Example 2: the explode round trip, done safely 

def with_parts(gdf, fn, *, id_col="_feature_id"):
    """Run fn on the exploded parts, then reassemble one row per input feature."""
    gdf = gdf.reset_index(drop=True).copy()
    gdf[id_col] = gdf.index

    parts = gdf.explode(index_parts=False)
    parts = fn(parts)                                   # per-part work happens here

    out = parts.dissolve(by=id_col, aggfunc="first").reset_index(drop=False)
    missing = set(gdf[id_col]) - set(out[id_col])
    if missing:
        print(f"⚠ {len(missing)} features lost every part during processing")
    return out.drop(columns=[id_col])

# drop sliver parts without changing the feature count
clean = with_parts(parcels, lambda p: p[p.geometry.area > 1.0])
assert len(clean) <= len(parcels)

The missing check is the part that matters: a feature whose every part was filtered away disappears entirely, and that is a row loss hiding inside a function written to prevent row gain.

Example 3: catching it in a test 

def test_cleaning_never_adds_rows(dirty_parcels):
    clean, _ = clean_parcels(dirty_parcels)
    assert len(clean) <= len(dirty_parcels)

def test_key_survives_cleaning(dirty_parcels):
    clean, _ = clean_parcels(dirty_parcels)
    assert clean["parcel_id"].is_unique

def test_multipart_features_stay_one_row(parcels_with_multipart):
    clean, _ = clean_parcels(parcels_with_multipart)
    assert len(clean) == len(parcels_with_multipart)

The third is the regression test for this whole page. Add a fixture with one MultiPolygon in it and the test fails the moment somebody inserts an explode.

Explanation

Two panels contrasting a layer where one row is one feature with one where rows are parts.
Both are legitimate. Problems start when a pipeline changes from one to the other without saying so.

The underlying issue is that a row in a GeoDataFrame does not have a fixed meaning. In one layer a row is a real-world feature; in another it is a part of one, or a piece produced by intersecting two layers. Both are legitimate models, and pandas has no opinion about which you are using.

Every operation that changes the count is silently changing which model you are in. explode moves from feature-rows to part-rows. overlay moves to piece-rows. sjoin moves to pair-rows. Each is documented and correct; none announce that the unit of the table has changed, and the columns come along unchanged, so a sum() written for the old unit still runs and returns a plausible wrong number.

That is why the primary-key assertion is worth more than the count assertion. Counts legitimately move up and down through a pipeline — filtering removes, clipping splits. But if parcel_id was unique on the way in and is not on the way out, the unit changed and nobody said so. One line catches every instance of this page's problem, regardless of which operation caused it.

The secondary trap — stale derived columns — follows from the same root. A stored area_m2 is a fact about the old geometry. After a clip, an explode, or a make_valid that changed a bowtie's area, it is a fact about a shape that no longer exists in the table. It still sums, it still maps, and it is wrong. The rule is unconditional: recompute anything derived from geometry after anything that changes geometry, and prefer computing it at the point of use over storing it at all.

Edge cases or notes

  • explode(index_parts=True) returns a MultiIndex, which then breaks .loc assignments written for a flat index. index_parts=False is almost always what you want.
  • dissolve sorts by the group key and returns it as the index. reset_index() or the key vanishes from the columns.
  • dissolve(aggfunc="first") silently picks a value for every non-key column. For sums, pass a dict of per-column aggregations.
  • overlay(how="union") produces rows for both inputs' leftovers, so it can more than double the count.
  • A sjoin with how="left" adds no rows when nothing matches but does when something matches twice — the asymmetry surprises people.
  • pd.concat of two GeoDataFrames with different CRS produces a frame with no CRS and no warning, on top of the extra rows.
  • make_valid returning a GeometryCollection breaks to_file for most drivers — filter to polygonal parts.
  • Row count is not the only unit that can change. A dissolve that merges 12,400 parcels into 32 wards is a deliberate unit change, and derived columns need the same treatment.

FAQ

Which operation is most likely to have added rows?

explode(), by a wide margin. A layer with 297 MultiPolygons averaging 2.5 parts adds about 450 rows, which is the size of gap most people report.

Why did my row count change only after a supplier update?

make_valid turns some invalid single-part polygons into multi-part ones. A later explode then produces more rows than it used to, without any code change.

Is explode wrong?

No. It is right when the parts are independent things and wrong when they are one thing. Add a stable id before exploding so you can dissolve back either way.

How do I stop a spatial join duplicating rows?

Deduplicate deterministically after the join, or aggregate the right-hand side. Never rely on file order for the tie-break — it varies between runs.

Why is my area total inflated after clipping?

The stored area column came through the clip unchanged, so a parcel split into two pieces contributes its full original area twice. Recompute area from geometry after the clip.

What is the single best check?

Assert the primary key is still unique. Counts legitimately change; a key going non-unique never does, and it catches every case on this page.

Should I store area and length columns at all?

Prefer computing them at the point of use. Stored values go stale the moment any geometry operation runs, and nothing warns.