🔧 Fix Data cleaning Geometry & topology Notebook By · Updated

My Cleaned Layer Has More Rows Than the Original: How to Fix It

On this page

Problem statement

Cleaning is supposed to remove things. The row count went up. Here is an ordinary cleaning function run over the ONS boundaries of the UK's 361 local authority districts (the data is set up under The data used on this page):

def clean_boundaries(gdf):
    gdf = gdf[gdf.geometry.notna() & ~gdf.geometry.is_empty]
    gdf = gdf.assign(geometry=gdf.geometry.make_valid())
    gdf = gdf.explode(index_parts=False)          # "single parts are easier to work with"
    return gdf[gdf.geometry.area > 0]


clean = clean_boundaries(districts)
print(f"{len(districts)} districts in, {len(clean):,} rows out")
print(f"LAD25CD still unique: {clean['LAD25CD'].is_unique}")
print(f"Shape__Area total: {districts['Shape__Area'].sum() / 1e6:,.0f} km² before, "
      f"{clean['Shape__Area'].sum() / 1e6:,.0f} km² after")
361 districts in, 5,389 rows out
LAD25CD still unique: False
Shape__Area total: 244,366 km² before, 30,918,931 km² after

Five thousand extra rows appeared. The district code is no longer a key, every total computed from a carried-over column is inflated many times over, and any join downstream will multiply again.

The cleaning code contains no concat, no join and no loop that appends. It still multiplied rows, because ordinary cleaning operations split one feature into several, and none of them warns.

Quick answer

Find which step changed the count, then decide whether the split was correct:

def trace_counts(gdf, steps):
    """Run each step in turn and report what it did to the row count."""
    n = len(gdf)
    print(f"{'start':24s} {n:>7,}")
    for name, fn in steps:
        gdf = fn(gdf)
        delta = len(gdf) - n
        flag = "  ←" if delta else ""
        print(f"{name:24s} {len(gdf):>7,}  {delta:+,}{flag}")
        n = len(gdf)
    return gdf


_ = trace_counts(districts, [
    ("drop null geometry", lambda g: g[g.geometry.notna()]),
    ("make_valid",         lambda g: g.assign(geometry=g.geometry.make_valid())),
    ("explode",            lambda g: g.explode(index_parts=False)),
    ("drop empty",         lambda g: g[~g.geometry.is_empty]),
])
start                        361
drop null geometry           361  +0
make_valid                   361  +0
explode                    5,389  +5,028  ←
drop empty                 5,389  +0
Operation Why it adds rows Fix
explode() one MultiPolygon → one row per part add a stable id before, dissolve after, or do not explode
overlay() / intersection() one input feature → one row per output piece aggregate back by the original id
sjoin() one row per matching pair deduplicate, or aggregate the right side
make_valid() a bow-tie becomes a MultiPolygon (row count same, parts change) check geom_type after
dissolve().explode() the classic round-trip that changes counts twice keep the dissolve key 
# the usual culprit, and the usual fix
districts["feature_id"] = districts["LAD25CD"]           # stable id BEFORE exploding
parts = districts.explode(index_parts=False)
back = parts.dissolve(by="feature_id", aggfunc="first").reset_index()
assert len(back) == len(districts)
print(f"{len(districts)} → {len(parts):,} parts → {len(back)} districts")
361 → 5,389 parts → 361 districts

Where the extra rows come from

Operations that change the row count and what one input row becomes. what one input row becomesone row becomesthe trapexplodeone row per part✕no way back without an idoverlayone row per piece✕stored area now double-countssjoinone row per match!boundary features match twicemake_validsame count, more parts!a later explode multipliesNone are bugs; all are surprises if the split was not expected spatialworkflow.io
None of these are bugs. All of them are surprises if you did not expect the split.

Find the step: explode, overlay, sjoin, make_valid

Each row-count symptom paired with its cause and fix. the count names the operationSymptomFixcount rose after explodeadd a feature id first, dissolveback aftercount rose after overlayexpected — but recompute derivedcolumnscount rose after sjoindeduplicate deterministically, oraggregatearea total inflatedstored area column is stale —recomputeprimary key no longer uniquethe unit of the table changedsilentlyAssert the key stays unique — it catches every row on this list spatialworkflow.io
The count tells you which operation; the id tells you whether it was correct.

1. Find the step, not the cause

Do not read the code looking for the bug. Instrument it — the count is the fastest diagnostic available, and it points at exactly one line.

def counted(fn, label):
    """Wrap any step so it reports what it did to the row count."""
    def wrapper(gdf, *a, **kw):
        before = len(gdf)
        out = fn(gdf, *a, **kw)
        if len(out) != before:
            print(f"  {label}: {before:,} → {len(out):,} ({len(out) - before:+,})")
        return out
    return wrapper


make_valid = counted(lambda g: g.assign(geometry=g.geometry.make_valid()), "make_valid")
explode = counted(lambda g: g.explode(index_parts=False), "explode")
_ = explode(make_valid(districts))
  explode: 361 → 5,389 (+5,028)

Only the step that changed the count speaks. Applied once, this turns "somewhere in cleaning" into one line.

2. explode() — the most common cause by far

print(districts.geom_type.value_counts().to_string())
parts = districts.explode(index_parts=False)
print(f"{len(parts):,} parts")
print(parts.groupby("LAD25NM").size().sort_values().tail(3).to_string())
Polygon         236
MultiPolygon    125
5,389 parts
LAD25NM
Argyll and Bute        701
Highland               722
Na h-Eileanan Siar    1417

explode is doing exactly what it says: one row per single-part geometry. A third of the districts are MultiPolygons, and the island councils hold most of the parts — Na h-Eileanan Siar alone becomes more than fourteen hundred rows. The question is whether you wanted that.

You want it when the parts are independent things — separate buildings recorded as one multi-feature, islands that should be counted individually.

You do not want it when the parts are one thing — a district with offshore islands, a parcel split by a road. Those must stay one row or every count and every sum is wrong.

If you exploded for a reason and need to get back:

districts = districts.reset_index(drop=True)
districts["feature_id"] = districts.index                 # stable, before exploding

parts = districts.explode(index_parts=False)
parts["part_area"] = parts.geometry.area                  # do the per-part work

parts = parts[parts["part_area"] > 10_000]                # drop rocks and skerries under 1 ha

back = parts.dissolve(by="feature_id", aggfunc="first").reset_index()
print(f"{len(districts)} → {len(parts):,} parts over 1 ha → {len(back)} districts")
361 → 2,215 parts over 1 ha → 361 districts

Without feature_id, the round trip is impossible — after exploding, nothing records which parts belonged together. See how to split multipart geometries.

3. overlay and intersection — one row per resulting piece

Leeds has 293,729 registered land titles in HM Land Registry's INSPIRE polygons and 33 electoral wards. Cutting the titles by ward:

parcels["area_m2"] = parcels.geometry.area                # a stored, derived column
clipped = gpd.overlay(parcels, wards, how="intersection")
print(f"{len(parcels):,} titles → {len(clipped):,} pieces")
293,729 titles → 295,765 pieces

That is correct behaviour: a title in two wards genuinely has two pieces, one in each. It becomes a bug when the next step sums area_m2 — a column carried over unchanged from before the split, so a title cut in two contributes its whole area twice.

print(f"stored area_m2:   {clipped['area_m2'].sum() / 1e6:,.1f} km²")
clipped["area_m2"] = clipped.geometry.area                # recompute after the split
print(f"recomputed:       {clipped['area_m2'].sum() / 1e6:,.1f} km²")
stored area_m2:   680.6 km²
recomputed:       535.7 km²

Any column derived from geometry must be recomputed after any operation that changes geometry. Stored Shape__Area, SHAPE_Length, perimeter — all of them are stale the moment a shape is clipped. (The recomputed total is also smaller than the titles' own total, because some Leeds titles run past the district boundary and the part outside no ward is dropped.)

4. sjoin — one row per matching pair

touching = gpd.sjoin(parcels, wards, predicate="intersects")
inside = gpd.sjoin(parcels, wards, predicate="within")
print(f"intersects: {len(touching):,} rows for {touching.index.nunique():,} titles")
print(f"within:     {len(inside):,} rows — {len(parcels) - len(inside):,} titles matched no ward at all")
intersects: 295,766 rows for 293,711 titles
within:     290,842 rows — 2,887 titles matched no ward at all

The two predicates fail in opposite directions. intersects gives a title that crosses a ward boundary one row per ward it touches. within can never match two wards — a polygon cannot lie inside two non-overlapping polygons at once — so it avoids duplicates by silently dropping every title that crosses a ward boundary or runs past the district edge. Neither is the right answer on its own. See spatial predicates explained for why, and duplicate rows from sjoin for the full treatment.

dupes = touching.index.duplicated(keep=False)
print(f"{dupes.sum():,} rows from {touching.index[dupes].nunique():,} multi-matched titles")

# resolve deterministically — never by file order: the ward holding most of the title
touching["overlap"] = [p.intersection(wards.geometry.iloc[i]).area
                       for p, i in zip(touching.geometry, touching["index_right"])]
one_each = (touching.sort_values(["INSPIREID", "overlap"], ascending=[True, False])
                    .loc[lambda d: ~d.index.duplicated(keep="first")])
print(f"{len(one_each):,} rows, INSPIREID unique: {one_each['INSPIREID'].is_unique}")
4,059 rows from 2,004 multi-matched titles
293,711 rows, INSPIREID unique: True

Or aggregate instead of choosing, when both matches are real:

wards_per_title = touching.groupby("INSPIREID")["WD24NM"].agg(lambda s: ";".join(sorted(set(s))))
print(wards_per_title[wards_per_title.str.contains(";")].head(3).to_string())
INSPIREID
25337855    Guiseley & Rawdon;Otley & Yeadon
25341670    Guiseley & Rawdon;Otley & Yeadon
25361677    Guiseley & Rawdon;Otley & Yeadon
Rows produced by joining 293,729 Leeds land titles to 33 wards with each predicate. rows wrong, against one row per matched title (293,711)intersects+2,055 — 2,004 titles repeatedwithin−2,869 — titles droppedlargest overlap0 — one row each02869 rowsThe two predicates miss in opposite directions, by thousands of rows each spatialworkflow.io
Against 293,711 matched titles (the other 18 lie wholly outside Leeds), each predicate is off by thousands in its own direction — both within 1% of the right total, which is why the key check, not the count, is the test.

5. make_valid — the count stays, the parts change

On these two layers make_valid has nothing to do — both are clean:

for name, layer in [("districts", districts), ("parcels", parcels)]:
    print(f"{name:10s} invalid: {(~layer.is_valid).sum()}")
districts  invalid: 0
parcels    invalid: 0

Supplier data is not always like that, and the mechanism is worth seeing on the smallest possible case — one self-intersecting "bow-tie" parcel:

from shapely import Polygon

bowtie = gpd.GeoDataFrame({"parcel_id": ["P1"]}, crs=27700,
                          geometry=[Polygon([(0, 0), (10, 10), (10, 0), (0, 10)])])
fixed = bowtie.assign(geometry=bowtie.geometry.make_valid())
print(f"before: {bowtie.geom_type.iloc[0]}, valid={bowtie.is_valid.iloc[0]}, {len(bowtie)} row")
print(f"after:  {fixed.geom_type.iloc[0]}, valid={fixed.is_valid.iloc[0]}, {len(fixed)} row")
print(f"explode after make_valid: {len(fixed.explode(index_parts=False))} rows")
before: Polygon, valid=False, 1 row
after:  MultiPolygon, valid=True, 1 row
explode after make_valid: 2 rows

make_valid does not add rows, but it turns single-part polygons into multi-part ones — so a later explode produces more rows than it would have before. That is why a pipeline that ran fine for months suddenly multiplies rows after a supplier sends slightly worse geometry. It can also return a GeometryCollection mixing polygons with lines or points; keep only the polygonal parts:

from shapely.geometry import GeometryCollection, MultiPolygon

def polygons_only(geom):
    if isinstance(geom, GeometryCollection):
        parts = [g for g in geom.geoms if g.geom_type in ("Polygon", "MultiPolygon")]
        return MultiPolygon([p for g in parts for p in getattr(g, "geoms", [g])]) if parts else None
    return geom

layer["geometry"] = layer.geometry.apply(polygons_only)

6. Assert the count you expect, at every boundary

def assert_row_count(gdf, expected, label=""):
    assert len(gdf) == expected, (
        f"{label}: expected {expected:,} rows, got {len(gdf):,} ({len(gdf) - expected:+,})"
    )


def assert_key_unique(gdf, key):
    dupes = gdf[key].duplicated()
    assert not dupes.any(), (
        f"{dupes.sum():,} duplicate {key} values, e.g. "
        f"{gdf.loc[dupes, key].head(3).tolist()}"
    )


try:
    assert_key_unique(clean, "LAD25CD")        # the output of the problem statement
except AssertionError as err:
    print(f"AssertionError: {err}")
AssertionError: 5,028 duplicate LAD25CD values, e.g. ['E06000001', 'E06000001', 'E06000001']

assert_key_unique is the more valuable of the two. Row counts legitimately change; a primary key going non-unique never should, and it is the condition that actually breaks the downstream join.

A cleaning function that cannot silently multiply

The data used on this page

Three real layers, all in British National Grid (EPSG:27700):

  • districts — the ONS December 2025 boundaries of the UK's 361 local authority districts (generalised, clipped to the coastline), with the service's own stored Shape__Area. A third of them are MultiPolygons.
  • wards — the 33 Leeds electoral wards, December 2024, full resolution.
  • parcels — HM Land Registry's INSPIRE Index Polygons for Leeds: 293,729 registered titles.
import sys

import geopandas as gpd
import pandas as pd
import pyogrio
import requests

ONS = "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services"
districts = gpd.read_file(f"{ONS}/Local_Authority_Districts_DEC_2025_Boundaries_UK_BGC/FeatureServer/0/query"
                          "?where=1%3D1&outFields=LAD25CD,LAD25NM,Shape__Area&outSR=27700&f=geojson")
districts = districts.set_crs(27700, allow_override=True).sort_values("LAD25CD").reset_index(drop=True)
districts.to_parquet("districts.parquet")

wards = gpd.read_file(f"{ONS}/Wards_December_2024_Boundaries_UK_BFC/FeatureServer/0/query"
                      "?where=LAD24NM%3D%27Leeds%27&outFields=WD24CD,WD24NM&outSR=27700&f=geojson")
wards = wards.set_crs(27700, allow_override=True).sort_values("WD24CD").reset_index(drop=True)

# HM Land Registry's download sets a cookie and redirects to itself, so use a session
reply = requests.Session().get("https://use-land-property-data.service.gov.uk/datasets/inspire/download/Leeds.zip",
                               timeout=600)
reply.raise_for_status()
with open("leeds_inspire.zip", "wb") as f:
    f.write(reply.content)
pyogrio.set_gdal_config_options({"GML_DOWNLOAD_SCHEMA": "NO"})   # the file's schema URL is unreachable
parcels = gpd.read_file("zip://leeds_inspire.zip!Land_Registry_Cadastral_Parcels.gml", columns=["INSPIREID"])

print(f"{len(districts)} districts, {len(wards)} wards, {len(parcels):,} titles")
361 districts, 33 wards, 293,729 titles

Example 1: a cleaning function that cannot silently multiply

%%writefile cleaning.py
from dataclasses import dataclass, field


@dataclass
class CleanReport:
    counts: list = field(default_factory=list)

    def step(self, label, before, after):
        self.counts.append({"step": label, "before": before, "after": after,
                            "delta": after - before})

    def __str__(self):
        lines = [f"{'step':22s} {'before':>8s} {'after':>8s} {'delta':>7s}"]
        for c in self.counts:
            mark = " ←" if c["delta"] else ""
            lines.append(f"{c['step']:22s} {c['before']:>8,} {c['after']:>8,} "
                         f"{c['delta']:>+7,}{mark}")
        return "\n".join(lines)


def clean_boundaries(gdf, *, key):
    report = CleanReport()

    def track(label, out):
        report.step(label, len(gdf), len(out))
        return out

    gdf = track("drop null geometry", gdf[gdf.geometry.notna()])
    gdf = track("drop empty geometry", gdf[~gdf.geometry.is_empty])
    gdf = track("make_valid", gdf.assign(geometry=gdf.geometry.make_valid()))
    gdf = track("polygons only", gdf[gdf.geom_type.isin(["Polygon", "MultiPolygon"])])

    assert gdf[key].is_unique, f"{key} is no longer unique after cleaning"
    return gdf, report
from cleaning import clean_boundaries

clean_districts, report = clean_boundaries(districts, key="LAD25CD")
print(report)
clean_parcels, report = clean_boundaries(parcels, key="INSPIREID")
print(report)
step                     before    after   delta
drop null geometry          361      361      +0
drop empty geometry         361      361      +0
make_valid                  361      361      +0
polygons only               361      361      +0
step                     before    after   delta
drop null geometry      293,729  293,729      +0
drop empty geometry     293,729  293,729      +0
make_valid              293,729  293,729      +0
polygons only           293,729  293,729      +0

Every step is accounted for — here, reassuringly, with nothing to report — and the key assertion means a multiplication cannot get past this function.

Example 2: the explode round trip, done safely

def with_parts(gdf, fn, *, id_col="_feature_id"):
    """Run fn on the exploded parts, then reassemble one row per input feature."""
    gdf = gdf.reset_index(drop=True).copy()
    gdf[id_col] = gdf.index

    parts = gdf.explode(index_parts=False)
    parts = fn(parts)                                   # per-part work happens here

    out = parts.dissolve(by=id_col, aggfunc="first").reset_index(drop=False)
    missing = set(gdf[id_col]) - set(out[id_col])
    if missing:
        print(f"⚠ {len(missing)} features lost every part during processing: "
              f"{sorted(gdf.loc[sorted(missing), 'LAD25NM'])[:3]}")
    return out.drop(columns=[id_col])


# drop parts under 20 km² without changing the district count... or so it looks
trimmed = with_parts(districts, lambda p: p[p.geometry.area > 20e6])
print(f"{len(districts)} districts in, {len(trimmed)} out")
⚠ 7 features lost every part during processing: ['City of London', 'Hackney', 'Hammersmith and Fulham']
361 districts in, 354 out

The missing check is the part that matters: a feature whose every part was filtered away disappears entirely — here the small urban districts whose single polygon is under the threshold — and that is a row loss hiding inside a function written to prevent row gain.

Example 3: catching it in a test

%%writefile test_cleaning.py
import geopandas as gpd
import pytest

from cleaning import clean_boundaries


@pytest.fixture
def districts():
    return gpd.read_parquet("districts.parquet")


@pytest.fixture
def districts_with_multipart(districts):
    return districts[districts.geom_type == "MultiPolygon"].head(20)


def test_cleaning_never_adds_rows(districts):
    clean, _ = clean_boundaries(districts, key="LAD25CD")
    assert len(clean) <= len(districts)


def test_key_survives_cleaning(districts):
    clean, _ = clean_boundaries(districts, key="LAD25CD")
    assert clean["LAD25CD"].is_unique


def test_multipart_features_stay_one_row(districts_with_multipart):
    clean, _ = clean_boundaries(districts_with_multipart, key="LAD25CD")
    assert len(clean) == len(districts_with_multipart)
!{sys.executable} -m pytest -q --no-header -p no:cacheprovider --color=no test_cleaning.py
...                                                                      [100%]
3 passed in 0.70s

The third is the regression test for this whole page. Its fixture holds twenty real multi-part districts, so the test fails the moment somebody inserts an explode into clean_boundaries.

A row has no fixed meaning

A layer where one row is one feature contrasted with one where rows are parts. what does one row mean here?One row = one feature✓parcel_id is unique✓count() counts parcels✓sum(area) is the total area✓a join gives one row back✓the key assertion holdsOne row = one part or pair✕parcel_id repeats✕count() counts parts✕sum(area) double-counts✕a join multiplies again✕the key assertion failsBoth are valid models — silently switching between them is not spatialworkflow.io
Both are legitimate. Problems start when a pipeline changes from one to the other without saying so.

The underlying issue is that a row in a GeoDataFrame does not have a fixed meaning. In one layer a row is a real-world feature; in another it is a part of one, or a piece produced by intersecting two layers. Both are legitimate models, and pandas has no opinion about which you are using.

Every operation that changes the count is silently changing which model you are in. explode moves from feature-rows to part-rows. overlay moves to piece-rows. sjoin moves to pair-rows. Each is documented and correct; none announce that the unit of the table has changed, and the columns come along unchanged, so a sum() written for the old unit still runs and returns a plausible wrong number.

That is why the primary-key assertion is worth more than the count assertion. Counts legitimately move up and down through a pipeline — filtering removes, clipping splits. But if parcel_id was unique on the way in and is not on the way out, the unit changed and nobody said so. One line catches every instance of this page's problem, regardless of which operation caused it.

The secondary trap — stale derived columns — follows from the same root. A stored area_m2 is a fact about the old geometry. After a clip, an explode, or a make_valid that changed a bowtie's area, it is a fact about a shape that no longer exists in the table. It still sums, it still maps, and it is wrong. The rule is unconditional: recompute anything derived from geometry after anything that changes geometry, and prefer computing it at the point of use over storing it at all.

MultiIndexes, dissolve order and union leftovers

  • explode(index_parts=True) returns a MultiIndex, which then breaks .loc assignments written for a flat index. index_parts=False is almost always what you want.

  • dissolve sorts by the group key and returns it as the index. reset_index() or the key vanishes from the columns.

  • dissolve(aggfunc="first") silently picks a value for every non-key column. For sums, pass a dict of per-column aggregations.

  • overlay(how="union") produces rows for both inputs' leftovers, so it can more than double the count.

  • A sjoin with how="left" adds no rows when nothing matches but does when something matches twice — the asymmetry surprises people.

  • pd.concat of two GeoDataFrames with different CRS is refused: GeoPandas 1.1 raises ValueError: Cannot determine common CRS for concatenation inputs. Older versions let it through, so old pipelines may already hold mixed frames.

  • make_valid returning a GeometryCollection is refused by the shapefile driver (Geometry type of 'Geometry Collection' not supported in shapefiles); GeoPackage and GeoJSON write it, which only moves the problem to whoever reads the file next. Filter to polygonal parts.

  • Row count is not the only unit that can change. A dissolve that merges 293,729 titles into 33 wards is a deliberate unit change, and derived columns need the same treatment.

FAQ

Which operation is most likely to have added rows?

explode(), by a wide margin. On this page the UK's 125 multi-part local authority districts held 5,153 parts between them, so one explode turned 361 rows into 5,389.

Why did my row count change only after a supplier update?

make_valid turns some invalid single-part polygons into multi-part ones. A later explode then produces more rows than it used to, without any code change.

Is explode wrong?

No. It is right when the parts are independent things and wrong when they are one thing. Add a stable id before exploding so you can dissolve back either way.

How do I stop a spatial join duplicating rows?

Deduplicate deterministically after the join, or aggregate the right-hand side. Never rely on file order for the tie-break — it varies between runs.

Why is my area total inflated after clipping?

The stored area column came through the clip unchanged, so a parcel split into two pieces contributes its full original area twice. Recompute area from geometry after the clip.

What is the single best check?

Assert the primary key is still unique. Counts legitimately change; a key going non-unique never does, and it catches every case on this page.

Should I store area and length columns at all?

Prefer computing them at the point of use. Stored values go stale the moment any geometry operation runs, and nothing warns.