How to Find and Remove Duplicate Geometries in GeoPandas

Duplicate geometries are one of the most common data-quality problems in spatial work, and they are easy to miss because they look fine on a map — two identical roads draw as one. This guide finds and removes exact duplicates, attribute duplicates, and near-duplicates in a GeoDataFrame with GeoPandas 1.x and Shapely 2.x, on duplicates that arose the ordinary way: by merging two neighbouring OpenStreetMap extracts.

Problem statement

Duplicate features creep into spatial datasets through ordinary, everyday operations. You usually do not introduce them on purpose, but they accumulate fast.

Common ways duplicates appear:

• Merged datasets. Concatenating tiles, regional extracts, or vendor deliveries that overlap at the edges produces the same feature twice — Geofabrik's Rutland and Leicestershire extracts share 108 roads that cross the county line.
• Repeated imports. Re-running an ETL job or loading the same file into a database table more than once.
• Joins and spatial joins. A one-to-many sjoin or attribute join multiplies rows so a single geometry appears many times.
• Digitising mistakes. An editor clicks "save" twice or copies a feature in place.
• Format round-trips. Reprojecting and re-importing shifts coordinates by nanometres, creating geometries that are equal in space but not byte-identical.

The problems they cause:

• Inflated counts. "How many roads?" returns the wrong number.
• Wrong aggregations. Sums of area, length, or population double-count.
• sjoin blowups. Duplicates on both sides of a spatial join multiply the output rows.
• Misleading statistics and maps. Stacked identical polygons distort choropleths and density analysis.

Quick answer

To remove exact duplicate geometries, drop_duplicates on the geometry — or on its WKB (well-known binary), which also works as a key alongside other columns. Verify the new count and export.

import geopandas as gpd
import pandas as pd

# Two neighbouring extracts, merged
gdf = pd.concat([gpd.read_file("data/rutland_roads.gpkg"), gpd.read_file("data/leicestershire_roads.gpkg")],
                ignore_index=True)

# Drop exact duplicate geometries
gdf["_wkb"] = gdf.geometry.to_wkb()
deduped = gdf.drop_duplicates(subset="_wkb").drop(columns="_wkb")

# Verify
print(f"Before: {len(gdf):,}  After: {len(deduped):,}  Removed: {len(gdf) - len(deduped)}")
print("same answer from the geometry column itself:", int(gdf.geometry.duplicated().sum()))

# Export
deduped.to_file("out/roads_deduped.gpkg", driver="GPKG")

Before: 106,467  After: 106,359  Removed: 108
same answer from the geometry column itself: 108

This catches identical geometries. For geometries that differ only by vertex order, normalise first; for ones that differ by a tolerance, see near-duplicates — the usual set_precision recipe is less reliable than it looks.

Find exact and near duplicates, then choose what to keep

Load the data

Read your layer, and check the row count and CRS up front, because CRS matters for any comparison you do later.

print(len(gdf), "rows |", gdf.crs, "|", gdf.geom_type.value_counts().to_dict())

106467 rows | EPSG:4326 | {'LineString': 106467}

Find exact duplicate geometries

import shapely

by_geometry = gdf.geometry.duplicated(keep=False)
by_wkb = gdf.geometry.to_wkb().duplicated(keep=False)
by_osm_id = gdf["osm_id"].duplicated(keep=False)
print("rows in a duplicate group - geometry:", int(by_geometry.sum()), "| WKB:", int(by_wkb.sum()),
      "| osm_id:", int(by_osm_id.sum()), "| all agree:", bool((by_geometry == by_wkb).all() and (by_wkb == by_osm_id).all()))

# the same road, digitised in the opposite direction
one = gdf.iloc[[0]].copy()
flipped = one.copy()
flipped["geometry"] = shapely.reverse(one.geometry.values)
pair = pd.concat([one, flipped])
print("reversed copy - geometry:", int(pair.geometry.duplicated().sum()),
      "| WKB:", int(pair.geometry.to_wkb().duplicated().sum()),
      "| normalised WKB:", int(pd.Series(shapely.to_wkb(shapely.normalize(pair.geometry.values))).duplicated().sum()),
      "| geom_equals:", bool(pair.geometry.iloc[0].equals(pair.geometry.iloc[1])))

rows in a duplicate group - geometry: 216 | WKB: 216 | osm_id: 216 | all agree: True
reversed copy - geometry: 0 | WKB: 0 | normalised WKB: 1 | geom_equals: True

gdf.geometry.duplicated() compares geometries by value, and on the merged extracts it found exactly what the WKB key and the OpenStreetMap ids found. What neither catches is the same shape stored differently: a road digitised in the other direction is the same line in space (equals says so) but a different sequence of vertices. shapely.normalize rewrites a geometry into a canonical vertex order and orientation, and after it the reversed copy is a duplicate.

Remove exact duplicates

By default drop_duplicates keeps the first occurrence; pass keep="last" to keep the last, or keep=False to drop every duplicated row entirely.

gdf["_wkb"] = shapely.to_wkb(shapely.normalize(gdf.geometry.values))
deduped = gdf.drop_duplicates(subset="_wkb", keep="first").drop(columns="_wkb")
print(f"Removed {len(gdf) - len(deduped)} duplicate geometries (vertex order ignored)")

Removed 108 duplicate geometries (vertex order ignored)

Handle duplicates that share geometry AND attributes vs geometry only

There are two different questions, and you must decide which you mean:

• Geometry only. Same shape, possibly different attribute values.
• Geometry AND attributes. The entire row is a duplicate, every column included — almost always safe to drop.

A spatial join is the everyday source of the first kind. Join Rutland's buildings to its wards, and a building that straddles a ward boundary comes back once per ward:

buildings = gpd.read_file("data/rutland_buildings.gpkg").to_crs(27700)
wards = gpd.read_file("data/rutland_wards.geojson").to_crs(27700)
joined = gpd.sjoin(buildings, wards[["WD24NM", "geometry"]], predicate="intersects").reset_index(drop=True)
joined["_wkb"] = joined.geometry.to_wkb()

full_row = joined.drop_duplicates(subset=["osm_id", "WD24NM", "_wkb"])
geometry_only = joined.drop_duplicates(subset="_wkb")
inside = joined["osm_id"].nunique()
print(f"{len(buildings):,} buildings in the extract, {inside:,} inside a Rutland ward -> {len(joined):,} rows after the join")
print("full-row dedupe keeps:", len(full_row), "| geometry-only dedupe keeps:", len(geometry_only))

5,513 buildings in the extract, 5,218 inside a Rutland ward -> 5,220 rows after the join
full-row dedupe keeps: 5220 | geometry-only dedupe keeps: 5218

Nothing in the join output is a full-row duplicate: the two buildings that straddle a ward boundary carry a different ward on each copy. Dropping on geometry alone silently keeps one ward and discards the other — a decision to make consciously, which is the aggregate step below.

Detect near-duplicates

Near-duplicates are geometries that are almost identical, off by tiny coordinate differences from rounding, reprojection, or separate digitising. The common recipe is to snap coordinates to a grid with shapely.set_precision and compare the snapped WKB. Here it is tested on Rutland's roads and a copy that went to British National Grid and back — a shift of at most 1.4 × 10⁻⁸ degrees, about a millimetre:

roads = gpd.read_file("data/rutland_roads.gpkg")
round_trip = roads.to_crs(27700).to_crs(4326)
both = pd.concat([roads, round_trip], ignore_index=True)
print("max shift:", f"{shapely.hausdorff_distance(roads.geometry.values, round_trip.geometry.values).max():.1e}", "degrees")
print("exact WKB duplicates:", int(both.geometry.to_wkb().duplicated().sum()), "of", len(roads))
for grid in (1e-9, 1e-7, 1e-6, 1e-5):
    key = pd.Series(shapely.to_wkb(shapely.normalize(shapely.set_precision(both.geometry.values, grid))))
    print(f"set_precision grid {grid:.0e}: {int(key.duplicated().sum()):>5} duplicates found")

max shift: 1.4e-08 degrees
exact WKB duplicates: 0 of 7292
set_precision grid 1e-09:     0 duplicates found
set_precision grid 1e-07:  7292 duplicates found
set_precision grid 1e-06:  3975 duplicates found
set_precision grid 1e-05:  6752 duplicates found

A grid finer than the shift found nothing, as expected. But the coarser grids did not find more and more: 1e-7 matched every road while 1e-6 matched barely half. Snapping is not a tolerance. Two coordinates a nanometre apart that fall either side of a grid line round to different values, and a road with many vertices needs only one such straddle to stop matching. Whether a pair straddles depends on where the grid lines happen to fall, which is why the counts jump about rather than rising with the grid size.

The reliable version compares candidate pairs within a tolerance:

def near_duplicates(gdf, tolerance, predicate="dwithin"):
    """Pairs of rows whose geometries are within `tolerance` of each other, vertex by vertex."""
    kwargs = {"distance": tolerance} if predicate == "dwithin" else {}
    left, right = gdf.sindex.query(gdf.geometry, predicate=predicate, **kwargs)
    candidate = left < right
    left, right = left[candidate], right[candidate]
    close = shapely.hausdorff_distance(gdf.geometry.values[left], gdf.geometry.values[right]) <= tolerance
    return list(zip(left[close], right[close]))


print(len(near_duplicates(both, 1e-7, predicate="intersects")), "pairs when the candidates come from an intersects query")
pairs = near_duplicates(both, tolerance=1e-7)
print(len(pairs), "near-duplicate pairs within 1e-7 degrees, candidates from dwithin")
drop = {j for i, j in pairs}
print(len(both), "->", len(both.drop(index=list(drop))), "rows")

1948 pairs when the candidates come from an intersects query
7292 near-duplicate pairs within 1e-7 degrees, candidates from dwithin
14584 -> 7292 rows

A spatial index finds the candidates, and the Hausdorff distance — the largest distance from any vertex of one to the other — decides. The candidates have to come from a distance query, dwithin, not intersects: a line shifted by a millimetre runs beside its copy without touching it, and an intersects query found barely a quarter of the pairs. With dwithin, every road found its round-tripped copy, with no grid lines involved.

Decide which row to keep / aggregate attributes

When duplicated geometries carry different attributes, you have three choices: keep the first, keep based on a rule, or aggregate the attributes into one row. dissolve groups by a key and keeps a valid geometry, with aggfunc controlling each attribute:

merged = joined.dissolve(by="_wkb", aggfunc={"osm_id": "first", "WD24NM": lambda s: " / ".join(sorted(set(s)))},
                         as_index=False).drop(columns="_wkb")
print(len(merged), "buildings;", int(merged["WD24NM"].str.contains(" / ").sum()), "of them now list two wards, e.g.",
      merged.loc[merged["WD24NM"].str.contains(" / "), "WD24NM"].iloc[0])

5218 buildings; 2 of them now list two wards, e.g. Cottesmore / Greetham

Verify the result

Never trust a dedupe blindly. Confirm that no duplicate keys remain and that the row count dropped by the expected amount.

check = shapely.to_wkb(shapely.normalize(deduped.geometry.values))
assert not pd.Series(check).duplicated().any(), "Duplicates still present!"
print("Rows before:", len(gdf), "| after:", len(deduped), "| unique geometries:", pd.Series(check).nunique())

Rows before: 106467 | after: 106359 | unique geometries: 106359

Deduplicating overlapping OpenStreetMap extracts

The data used on this page

Roads and buildings from Geofabrik's OpenStreetMap extracts for Rutland and for Leicestershire, both of 1 January 2026 (© OpenStreetMap contributors, ODbL), and Rutland's wards from the ONS December 2024 boundaries (OGL). Geofabrik keeps a way that crosses the county line in both extracts, which is where the duplicates come from.

from pathlib import Path

import geopandas as gpd
import pandas as pd
import requests
import shapely

for folder in ("data", "out"):
    Path(folder).mkdir(exist_ok=True)
session = requests.Session()
for county in ("rutland", "leicestershire"):
    if not Path(f"data/{county}_roads.gpkg").exists():
        archive = Path(f"data/{county}.shp.zip")
        archive.write_bytes(session.get(f"https://download.geofabrik.de/europe/united-kingdom/england/"
                                        f"{county}-260101-free.shp.zip", timeout=600).content)
        gpd.read_file(f"/vsizip/{archive}/gis_osm_roads_free_1.shp").to_file(f"data/{county}_roads.gpkg")
        if county == "rutland":
            gpd.read_file(f"/vsizip/{archive}/gis_osm_buildings_a_free_1.shp").to_file("data/rutland_buildings.gpkg")
if not Path("data/rutland_wards.geojson").exists():
    Path("data/rutland_wards.geojson").write_bytes(session.get(
        "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/"
        "Wards_December_2024_Boundaries_UK_BFC/FeatureServer/0/query",
        params={"where": "LAD24CD='E06000017'", "outFields": "WD24NM", "outSR": 4326, "f": "geojson"}, timeout=120).content)
print({p.name: len(gpd.read_file(p)) for p in sorted(Path("data").glob("*.gpkg"))})

{'leicestershire_roads.gpkg': 99175, 'rutland_buildings.gpkg': 5513, 'rutland_roads.gpkg': 7292}

Example 1: Count duplicate geometries without removing them

wkb = gdf.geometry.to_wkb()
print(f"{int(wkb.duplicated(keep=False).sum())} rows participate in duplicates")
print(f"{int(wkb.duplicated(keep='first').sum())} rows are redundant and can be removed")
print("by road class:", gdf.loc[wkb.duplicated(), "fclass"].value_counts().head(4).to_dict())

216 rows participate in duplicates
108 rows are redundant and can be removed
by road class: {'service': 24, 'bridleway': 20, 'footway': 19, 'unclassified': 16}

Example 2: Dedupe on a subset of columns plus geometry

gdf["_wkb"] = gdf.geometry.to_wkb()
clean = gdf.drop_duplicates(subset=["osm_id", "fclass", "_wkb"], keep="first").drop(columns="_wkb")
print(len(gdf), "->", len(clean))

106467 -> 106359

A row is a duplicate only if the id, the class and the geometry all match — the safe form when two sources might describe the same shape differently.

Example 3: Why a vectorised key beats a loop

import time

wkb_all = gdf.geometry.to_wkb()
sample = pd.concat([gdf.geometry[wkb_all.duplicated(keep=False)], gdf.geometry[~wkb_all.duplicated(keep=False)].iloc[:1284]])
start = time.perf_counter()
loop_pairs = sum(sample.iloc[i].equals(sample.iloc[j]) for i in range(len(sample)) for j in range(i + 1, len(sample)))
loop_seconds = time.perf_counter() - start
start = time.perf_counter()
key_dupes = int(pd.Series(shapely.to_wkb(shapely.normalize(sample.values))).duplicated().sum())
key_seconds = time.perf_counter() - start
print(f"pairwise .equals over {len(sample):,} rows: {loop_pairs} pairs in {loop_seconds:.1f} s")
print(f"normalised WKB key:                   {key_dupes} duplicates in {key_seconds * 1000:.1f} ms")

pairwise .equals over 1,500 rows: 108 pairs in 6.4 s
normalised WKB key:                   108 duplicates in 1.6 ms

The loop compares every pair — 1,124,250 comparisons for 1,500 rows, growing with the square of the count — while the key is one vectorised pass and a hash.

What drop_duplicates actually compares

What drop_duplicates compares. GeoPandas compares geometries by value, so identical coordinates in identical order are duplicates whether you use the geometry column or its WKB. The WKB column earns its place as a key you can combine with other columns in subset=, and because normalize must be applied to it for order-independence.

What to_wkb() does. Well-known binary is a standard, deterministic serialisation of a geometry's type and coordinates. Two geometries with the same type, same vertices, in the same order produce byte-identical WKB. Bytes are hashable and compare cheaply, which is exactly what pandas needs to detect duplicates fast.

What normalize() does. shapely.normalize rewrites a geometry into a canonical form: rings get a consistent orientation, vertices a consistent starting point, lines a consistent direction, and multi-part geometries a consistent component order. Run it before to_wkb() and geometries that were equal in space but stored differently produce identical bytes.

What set_precision() does, and does not. shapely.set_precision(geom, grid_size) snaps coordinates to a regular grid. Coordinates in the same grid cell collapse to the same value — but coordinates a hair apart in neighbouring cells do not, so snapping catches most near-duplicates and misses some, unpredictably, as the round-trip test showed. For a guarantee, compare within a tolerance.

Exact vs near duplicates. An exact duplicate is the same shape down to the vertex (modulo order, after normalize). A near duplicate is the same shape only within some tolerance, and needs that tolerance chosen deliberately — too large, and genuinely distinct features merge.

Duplicates the keys miss

MultiPolygon vs Polygon equivalence

A single Polygon and a MultiPolygon containing only that polygon describe the same area but have different geometry types and different WKB, and they are not duplicates to any of the keys above. Explode multiparts, or promote everything to multi, before building the key if you need them to match.

CRS must match before comparing

Coordinate values only mean the same thing within the same CRS. Concatenate layers in different CRSs and identical real-world features have different coordinates; a metre-based tolerance is meaningless across mixed CRSs. Reproject everything to one CRS first — the round-trip test shows even that leaves nanometre differences.

Duplicates with different attributes need a keep/aggregate decision

When the same geometry appears with conflicting attribute values — the buildings on ward boundaries — dropping duplicates silently discards data. Decide whether to keep the first, keep by a rule (sort, then keep="first"), or aggregate with dissolve.

Empty geometries

empties = gpd.GeoSeries([shapely.Point(), shapely.Point(), shapely.Polygon(), shapely.Polygon(),
                         shapely.from_wkt("LINESTRING EMPTY"), shapely.Point(0, 0)])
print("duplicates among 5 empties of 3 types and 1 point - WKB:", int(empties.to_wkb().duplicated().sum()),
      "| GeoSeries.duplicated:", int(empties.duplicated().sum()),
      "| normalised WKB:", int(pd.Series(shapely.to_wkb(shapely.normalize(empties.values))).duplicated().sum()))

duplicates among 5 empties of 3 types and 1 point - WKB: 2 | GeoSeries.duplicated: 2 | normalised WKB: 2

Empty geometries have no coordinates to tell them apart: empties of the same type collapse into one another, empties of different types do not, and none of that is what you meant. Remove them before deduplicating — see removing null and empty geometries.

Internal links

• How to Fix Invalid Geometries in Python (GeoPandas)
• How to Remove Null and Empty Geometries in GeoPandas
• How to Perform a Spatial Join in Python (GeoPandas)
• How to Read a Shapefile in Python with GeoPandas

FAQ

Why does drop_duplicates not catch my duplicate geometries?

GeoPandas does compare geometries by value — on two merged OpenStreetMap extracts, gdf.geometry.duplicated() found the same 108 duplicates as a WKB key and the OSM ids. What it misses is the same shape stored differently: a road digitised in the other direction, or a copy that went through a reprojection and came back a nanometre off. Normalise with shapely.normalize for the first, and compare within a tolerance for the second.

How do I remove near-duplicate polygons?

Find candidate pairs with a spatial index query by distance — gdf.sindex.query(gdf.geometry, predicate="dwithin", distance=tol) — and keep the pairs whose Hausdorff distance is within the tolerance. The popular alternative, snapping with set_precision and comparing WKB, is not a tolerance: on 7,292 reprojected roads it matched all of them at one grid size and barely half at a coarser one, because nearby coordinates can round to different grid points.

What is the difference between exact and near duplicates?

Exact duplicates have identical geometry down to the vertex (after normalising for order); near duplicates are the same shape only within a tolerance. Use normalize + to_wkb for exact, and a distance query plus a Hausdorff check for near. The near-duplicate case requires you to choose how much coordinate difference is acceptable.

Should I keep the first row or aggregate attributes?

If the duplicated rows are truly identical, keep the first — it loses nothing. If they share a geometry but carry different attribute values, decide whether one row is authoritative (sort then keep="first") or whether the values should be combined with dissolve/groupby using an aggfunc. Aggregating prevents silently throwing away real data.

Why are MultiPolygons and Polygons not matching as duplicates?

They have different geometry types and therefore different WKB, even when they cover the same area. A WKB comparison treats type as part of the identity. Explode multiparts or otherwise normalise geometry types before building your dedupe key if you need them to match.

How do I count duplicates without removing them?

Build a WKB (or snapped WKB) key and call .duplicated(keep=False).sum() to count every row involved in a duplicate, or .duplicated(keep="first").sum() to count only the redundant rows that would be removed. This lets you audit a dataset before deciding how to clean it.

Is to_wkb faster than comparing geometries in a loop?

Yes, by orders of magnitude. On 1,500 roads a pairwise .equals() loop took 6.4 s to find 108 duplicate pairs; a normalised WKB key found the same 108 in under 2 ms, and the loop grows with the square of the row count.