How to Merge Near-Duplicate Features in a Spatial Dataset #

Problem statement #

Deduplicating on exact geometry found 44 rows. The layer still has thousands of duplicates.

gdf.geometry.to_wkb().duplicated().sum()      # 44 — the easy ones
len(gdf)                                       # 18,204
gdf["name"].nunique()                          # 11,880

Six thousand rows share a name with another row. Some are genuine — two branches of the same chain — and some are the same building recorded twice, once from a survey and once from an address import, with coordinates two metres apart and the name spelled slightly differently.

id     name                          x         y
1042   "Royal Infirmary"          325113.2  674881.9
7781   "Royal Infirmary "         325115.0  674880.4     ← trailing space, 2.3 m away
9903   "ROYAL INFIRMARY (A&E)"    325112.8  674882.1     ← same place, different label

Exact deduplication cannot see these. drop_duplicates on the name cannot either — and even if it could, deciding which row survives, and what happens to the attributes of the ones that do not, is the actual work.

Quick answer #

Near-duplicate detection is a three-part decision: how close in space, how similar in text, and which record wins.

import geopandas as gpd
import pandas as pd
from rapidfuzz import fuzz

TOLERANCE_M = 25        # candidates must be within this
NAME_SCORE  = 85        # and this similar by name

def find_near_duplicates(gdf, name_col="name", tol=TOLERANCE_M, score=NAME_SCORE):
    # 1. spatial candidates — the index makes this cheap
    pairs = gpd.sjoin_nearest(
        gdf[[name_col, "geometry"]], gdf[[name_col, "geometry"]],
        max_distance=tol, distance_col="dist", how="inner",
    ).reset_index(names="left_i")
    pairs = pairs[pairs.left_i < pairs.index_right]      # each pair once, no self-match

    # 2. textual confirmation
    pairs["score"] = [
        fuzz.token_sort_ratio(str(a).strip().lower(), str(b).strip().lower())
        for a, b in zip(pairs[f"{name_col}_left"], pairs[f"{name_col}_right"])
    ]
    return pairs[pairs["score"] >= score].sort_values("score", ascending=False)

dupes = find_near_duplicates(places)
print(f"{len(dupes)} near-duplicate pairs")

The tolerance is the one that decides everything. Too tight and the duplicates survive; too loose and two genuinely different shops on the same street merge into one.

Exact versus near duplicates #

Step-by-step solution #

1. Remove exact duplicates first — they are free #

before = len(gdf)
gdf = gdf.loc[~gdf.geometry.to_wkb().duplicated()]
print(f"{before - len(gdf)} exact geometry duplicates removed")

to_wkb() compares the binary representation, so it catches identical shapes regardless of how they were built. Note it is sensitive to vertex order — two rings describing the same square from different start points are not equal. For that, normalize() first:

gdf = gdf.assign(geometry=gdf.geometry.normalize())    # canonical vertex order
gdf = gdf.loc[~gdf.geometry.to_wkb().duplicated()]

2. Block by space before comparing text #

Comparing every pair of 18,204 rows is 165 million comparisons. Using the spatial index to propose candidates first reduces it to a few thousand.

pairs = gpd.sjoin_nearest(
    gdf, gdf, max_distance=25, distance_col="dist", how="inner"
).reset_index(names="left_i")
pairs = pairs[pairs.left_i < pairs.index_right]        # dedupe the pair list itself

For polygons, overlap fraction is often a better blocker than distance:

def overlap_fraction(a, b):
    inter = a.intersection(b).area
    return inter / min(a.area, b.area) if min(a.area, b.area) else 0.0

pairs["overlap"] = [
    overlap_fraction(gdf.geometry.iloc[a], gdf.geometry.iloc[b])
    for a, b in zip(pairs.left_i, pairs.index_right)
]
candidates = pairs[pairs["overlap"] > 0.8]      # 80% of the smaller shape

Two buildings digitised twice overlap almost completely. Two adjacent buildings barely overlap at all. That ratio separates them far more reliably than centroid distance, which is fooled by long thin shapes.

3. Score the text, and choose the right scorer #

from rapidfuzz import fuzz

a, b = "Royal Infirmary", "ROYAL INFIRMARY (A&E)"
fuzz.ratio(a.lower(), b.lower())              # 76  — punished by the suffix
fuzz.partial_ratio(a.lower(), b.lower())      # 100 — one contains the other
fuzz.token_sort_ratio(a.lower(), b.lower())   # 76  — order-insensitive
fuzz.token_set_ratio(a.lower(), b.lower())    # 100 — ignores extra tokens

• token_sort_ratio — right when word order varies: "Smith & Sons" vs "Sons & Smith".
• token_set_ratio — right when one name has extra words: "Royal Infirmary" vs "Royal Infirmary (A&E)". Beware: it also scores "High Street" and "High Street West" at 100.
• ratio — strictest, right for codes and references.

Normalise before scoring, or the scorer spends its budget on punctuation:

import re

def normalise_name(s):
    s = str(s).lower().strip()
    s = re.sub(r"\b(ltd|limited|plc|the|and|&)\b", " ", s)
    s = re.sub(r"[^\w\s]", " ", s)
    return re.sub(r"\s+", " ", s).strip()

See how to fuzzy-match place names for the matching side of this.

4. Cluster — duplicates come in groups, not pairs #

Three records of the same hospital produce three pairs. Merging pairwise gives the wrong answer; you need the group.

import networkx as nx

def cluster_pairs(pairs, n_rows, left="left_i", right="index_right"):
    g = nx.Graph()
    g.add_nodes_from(range(n_rows))
    g.add_edges_from(zip(pairs[left], pairs[right]))
    return list(nx.connected_components(g))

clusters = [c for c in cluster_pairs(candidates, len(gdf)) if len(c) > 1]
print(f"{len(clusters)} clusters covering {sum(len(c) for c in clusters)} rows")

Without networkx, a union-find is a dozen lines:

def union_find(pairs, n):
    parent = list(range(n))
    def find(x):
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x
    for a, b in pairs:
        ra, rb = find(a), find(b)
        if ra != rb:
            parent[max(ra, rb)] = min(ra, rb)      # deterministic
    groups = {}
    for i in range(n):
        groups.setdefault(find(i), []).append(i)
    return [v for v in groups.values() if len(v) > 1]

Transitivity is worth watching: A matches B, B matches C, but A and C may be 48 m apart. Connected components will merge all three. If that is wrong for your data, cap the cluster diameter:

clusters = [
    c for c in clusters
    if gdf.geometry.iloc[list(c)].union_all().convex_hull.length < 200
]

5. Decide survivorship explicitly #

This is the part people skip, and it is the part that determines whether the result is correct.

def choose_survivor(group: gpd.GeoDataFrame) -> pd.Series:
    """Most complete record wins; ties break on most recent, then lowest id."""
    scored = group.assign(_completeness=group.notna().sum(axis=1))
    return scored.sort_values(
        ["_completeness", "surveyed", "id"], ascending=[False, False, True]
    ).iloc[0]

Three common rules, all defensible, none universal:

And the fourth option, which is often the best: merge rather than choose.

def merge_group(group: gpd.GeoDataFrame) -> pd.Series:
    """Take the best geometry, then fill each attribute from the first row that has it."""
    ordered = group.sort_values(["surveyed", "id"], ascending=[False, True])
    merged = ordered.iloc[0].copy()
    for col in group.columns:
        if col == "geometry" or pd.notna(merged[col]):
            continue
        filled = ordered[col].dropna()
        if len(filled):
            merged[col] = filled.iloc[0]
    merged["merged_from"] = ",".join(str(i) for i in sorted(group["id"]))
    merged["merged_count"] = len(group)
    return merged

merged_from is the line that makes this reversible. Six months later, "why does this record say 1998?" has an answer.

Code examples #

Example 1: the complete deduplication #

import geopandas as gpd
import pandas as pd
import networkx as nx
from rapidfuzz import fuzz

def deduplicate(gdf, *, name_col="name", tol=25, score=85, id_col="id"):
    gdf = gdf.reset_index(drop=True)
    report = {"in": len(gdf)}

    # exact first
    exact = gdf.geometry.normalize().to_wkb().duplicated()
    report["exact_removed"] = int(exact.sum())
    gdf = gdf.loc[~exact].reset_index(drop=True)

    # spatial candidates
    pairs = gpd.sjoin_nearest(
        gdf[[name_col, "geometry"]], gdf[[name_col, "geometry"]],
        max_distance=tol, distance_col="dist", how="inner",
    ).reset_index(names="a")
    pairs = pairs[pairs.a < pairs.index_right].rename(columns={"index_right": "b"})

    # text confirmation
    pairs["score"] = [
        fuzz.token_sort_ratio(normalise_name(x), normalise_name(y))
        for x, y in zip(pairs[f"{name_col}_left"], pairs[f"{name_col}_right"])
    ]
    matched = pairs[pairs["score"] >= score]
    report["candidate_pairs"] = len(pairs)
    report["matched_pairs"] = len(matched)

    # cluster
    g = nx.Graph()
    g.add_nodes_from(range(len(gdf)))
    g.add_edges_from(zip(matched.a, matched.b))
    clusters = [sorted(c) for c in nx.connected_components(g) if len(c) > 1]
    report["clusters"] = len(clusters)

    # merge
    keep_singletons = gdf.drop(index=[i for c in clusters for i in c])
    merged_rows = [merge_group(gdf.iloc[c]) for c in clusters]
    out = gpd.GeoDataFrame(
        pd.concat([keep_singletons, gpd.GeoDataFrame(merged_rows, crs=gdf.crs)]),
        crs=gdf.crs,
    ).reset_index(drop=True)

    report["out"] = len(out)
    report["rows_merged_away"] = report["in"] - report["exact_removed"] - report["out"]
    return out, report, matched

clean, report, pairs = deduplicate(places)
print(report)
# {'in': 18204, 'exact_removed': 44, 'candidate_pairs': 3117, 'matched_pairs': 1204,
#  'clusters': 981, 'out': 17021, 'rows_merged_away': 1139}

Example 2: tuning the threshold with evidence, not intuition #

def threshold_sweep(pairs, lo=70, hi=100, step=5):
    """How many pairs match at each threshold — and what they look like."""
    for t in range(lo, hi + 1, step):
        hits = pairs[pairs["score"] >= t]
        sample = hits.nsmallest(2, "score")[["name_left", "name_right", "score", "dist"]]
        print(f"\nthreshold {t}: {len(hits):,} pairs")
        for r in sample.itertuples():
            print(f"    {r.score:3.0f}  {r.dist:5.1f}m  {r.name_left!r} ↔ {r.name_right!r}")

threshold 80: 1,842 pairs
     80    18.2m  'High Street' ↔ 'High Street West'          ← false positive
threshold 85: 1,204 pairs
     85     3.1m  'Royal Infirmary' ↔ 'Royal Infirmary A&E'   ← correct
threshold 90:   712 pairs
     90     1.2m  'Tesco Express' ↔ 'Tesco Express '          ← correct, but 492 lost

Reading the weakest matches at each level is how you pick a threshold. The number alone tells you nothing; the pair at the boundary tells you everything.

Example 3: reviewing before committing #

def export_for_review(gdf, clusters, path):
    """One row per cluster member, grouped, so a human can scan it in QGIS."""
    rows = []
    for cid, members in enumerate(clusters):
        sub = gdf.iloc[members]
        for r in sub.itertuples():
            rows.append({
                "cluster": cid, "id": r.id, "name": r.name,
                "members": len(members),
                "spread_m": round(sub.geometry.union_all().convex_hull.length, 1),
                "geometry": r.geometry,
            })
    gpd.GeoDataFrame(rows, crs=gdf.crs).to_file(path, driver="GPKG")

Sort by spread_m descending and the doubtful clusters float to the top — those are the ones where transitivity has chained together things that are not the same.

Explanation #

Near-duplicate detection is a record linkage problem, and it inherits record linkage's central fact: there is no threshold that produces zero errors. Tighten it and real duplicates survive; loosen it and distinct things merge. The only question is which error is more expensive for the use you have in mind.

That asymmetry usually has a clear answer. For a mailing list, a false merge means one letter instead of two — annoying. For an asset register, a false merge means a bridge disappears from the maintenance schedule — serious. When a false merge is expensive, set the threshold high and flag the rest for review rather than tuning until the count looks right.

The spatial dimension is what makes this tractable compared with plain text deduplication. Two identically named shops in different cities are not duplicates, and a purely textual matcher cannot tell. Blocking on proximity first uses the spatial index to reduce a quadratic problem to a nearly linear one, and simultaneously removes the largest class of false positives for free. It is both the performance fix and the accuracy fix.

Clustering matters more than it appears. Duplicates arrive in groups — the same hospital in a survey, an address file and an OSM extract — and pairwise merging processes A+B, then discovers C matches a row that no longer exists. Building the graph first and merging each connected component once is the only approach that terminates with the right answer.

Finally, survivorship is where the information is lost. Every other step is reversible in principle; choosing one record and discarding two is not, unless you record what was discarded. That is what merged_from and merged_count are for, and why writing the merged-away rows to a side file costs nothing and answers every later question.

Edge cases or notes #

• sjoin_nearest needs GeoPandas 0.10+ and a projected CRS for max_distance to be in metres.
• to_wkb().duplicated() is vertex-order sensitive. normalize() first for a canonical order.
• Transitivity can over-merge. Cap cluster diameter or use a stricter threshold when chains appear.
• token_set_ratio scores "High Street" and "High Street West" at 100. It is the most permissive scorer and needs the highest threshold.
• rapidfuzz is a drop-in replacement for fuzzywuzzy and is roughly an order of magnitude faster.
• Merging polygons changes area. Decide whether the survivor keeps its own geometry or the union of the group — they are different answers.
• Deduplication is not idempotent by default. Run it twice and clusters can shift if tie-breaks depend on row order; sort explicitly.
• Keep the removed rows. A rejected_duplicates.gpkg beside the output costs nothing.

Internal links #

• How to find and remove duplicate geometries in GeoPandas — the exact-match case
• How to fuzzy-match place names when joining spatial data — the same scoring, for joining rather than merging
• Repair, reject or flag? Choosing what cleaning should do — survivorship as a cleaning decision
• Spatial indexes explained: R-trees and why spatial joins are fast — why blocking works
• How to find the nearest point in GeoPandas — sjoin_nearest in general
• GeoPandas spatial join returns duplicate rows — duplicates created by a join rather than found in data
• How to clean and normalise attribute columns in a GeoDataFrame — normalising before scoring
• The Python GIS data cleaning checklist — where this step belongs

FAQ #

What distance tolerance should I use? #

It depends on how the data was captured: 10 m for consumer GPS, 25 m for geocoded addresses, 5 m for digitised buildings. Start from the capture method, not from the result.

Which fuzzy scorer is best for place names? #

token_sort_ratio at around 85. It handles word-order differences without being as permissive as token_set_ratio, which scores any name containing another at 100.

Why cluster instead of merging pairs? #

Duplicates come in groups. Pairwise merging processes A+B and then finds C matching a row that no longer exists. Connected components handle the group in one pass.

How do I stop two different shops on one street merging? #

Tighten the distance tolerance, use overlap fraction rather than centroid distance for polygons, and require a stricter name score. Then review the weakest matches.

Which record should survive? #

Whichever rule you can defend: most complete, most recent, or most authoritative. Better still, merge — take the best geometry and fill each attribute from the first record that has it.

How do I make it reproducible? #

Sort before every tie-break and never rely on file order. Deduplication that depends on row order gives different answers on different machines.

Should I delete the duplicates? #

Merge them and record what was merged, with a merged_from column and a side file of the removed rows. Deletion without a record cannot be reviewed.