How to Merge Near-Duplicate Features in a Spatial Dataset

On this page

Problem statement

Two sources describe the same places. Leeds City Council's food hygiene register (via the Food Standards Agency) lists every food business it inspects; OpenStreetMap's mappers have drawn the cafés, pubs and takeaways they walk past. Put them in one layer and the same café is in it twice (the setup is under The data used on this page):

print(places["source"].value_counts().to_string())
print(places[places["name"].str.lower().isin(["friends of ham", "black sheep coffee"])]
      .sort_values("name")[["source", "name", "fhrsid", "osm_id"]].to_string())

source
fsa    5803
osm    2125
     source                name   fhrsid       osm_id
316     fsa  Black Sheep Coffee  1904309          NaN
317     fsa  Black Sheep Coffee  1904308          NaN
5973    osm  Black Sheep Coffee      NaN    333799014
7824    osm  Black Sheep Coffee      NaN  11144743506
3869    fsa      Friends Of Ham  1070612          NaN
6317    osm      Friends of Ham      NaN   2157985584

Both sources also list two separate Black Sheep Coffee branches, which is the other half of the problem: a matching name is not a matching place. The obvious fix is to drop duplicate geometries. It finds plenty:

same_point = places.geometry.to_wkb().duplicated()
print(f"{same_point.sum():,} rows share their exact coordinates with an earlier row")

1,745 rows share their exact coordinates with an earlier row

Almost none of them are duplicates. The FSA places each business at its address point, so every unit in a shopping centre or an office block gets the same coordinates, while the real duplicates — one record from each source — are metres to tens of metres apart and spelled differently. Exact deduplication deletes the wrong rows and keeps the right ones.

Quick answer

Near-duplicate detection is a three-part decision: how close in space, how similar in text, and which record wins.

import re

from rapidfuzz import fuzz

TOLERANCE_M = 50        # candidates must be within this
NAME_SCORE = 85         # and this similar by name


def normalise_name(s):
    s = str(s).lower().strip()
    s = re.sub(r"\b(ltd|limited|plc|the|and)\b|&", " ", s)
    s = re.sub(r"[^\w\s]", " ", s)
    return re.sub(r"\s+", " ", s).strip()


def find_near_duplicates(gdf, name_col="name", tol=TOLERANCE_M, score=NAME_SCORE):
    # 1. spatial candidates — every pair within tol, from the spatial index
    a, b = gdf.sindex.query(gdf.geometry.values, predicate="dwithin", distance=tol)
    a, b = a[a < b], b[a < b]                              # each pair once, no self-match
    pairs = pd.DataFrame({"a": a, "b": b,
                          "dist": gdf.geometry.values[a].distance(gdf.geometry.values[b])})

    # 2. textual confirmation
    names = gdf[name_col].map(normalise_name).values
    pairs["score"] = [fuzz.token_sort_ratio(names[i], names[j]) for i, j in zip(a, b)]
    return pairs[pairs["score"] >= score].reset_index(drop=True)


dupes = find_near_duplicates(places)
src = places["source"].values
print(f"{len(dupes)} near-duplicate pairs:")
print(pd.Series([f"{src[i]}–{src[j]}" for i, j in zip(dupes["a"], dupes["b"])]).value_counts().to_string())

684 near-duplicate pairs:
fsa–osm    653
fsa–fsa     26
osm–osm      5

Most pairs are one FSA record and one OSM record — the same business seen by both sources. The same-source pairs deserve a second look before anything merges them: two FSA registrations with the same name at the same address are usually two kitchens in one hospital or one venue, not a typing error.

Decision	Question it answers	On this page
spatial tolerance	how far apart can the same thing be recorded?	50 m — address points against mapped entrances
similarity threshold	how differently can the same thing be spelled?	85 on normalised names
survivorship rule	which record wins, and what happens to the rest?	merge: keep both sources' ids

The tolerance is the one that decides everything. Too tight and the duplicates survive; too loose and two branches of the same chain on one street merge into one — example 2 shows exactly where that starts.

Exact versus near duplicates

Exact-geometry deduplication gets the second row wrong and never sees the third; the fourth is the risk of solving it too aggressively.

Exact duplicates, spatial blocking, text scores, clusters

Blocking before scoring. Comparing every pair of 7,928 rows is 31 million comparisons.

1. Remove exact duplicates first — but define "exact" properly

key = places.geometry.to_wkb()
groups = places.groupby(key)
sizes, distinct_names = groups.size(), groups["name"].nunique()
print(f"{(sizes > 1).sum()} shared locations holding {sizes[sizes > 1].sum():,} rows; "
      f"{(distinct_names[sizes > 1] > 1).sum()} of them hold more than one name")
print("the busiest:", sorted(places.loc[key == sizes.idxmax(), "name"])[:6], "…")

exact = pd.Series(list(zip(key, places["name"].map(normalise_name)))).duplicated()
print(f"same point AND same normalised name: {exact.sum()} rows")
places = places.loc[~exact.values].reset_index(drop=True)

899 shared locations holding 2,644 rows; 893 of them hold more than one name
the busiest: ["Batch'd", 'Bella Italia Also T/a Bird Box, Stacks-Burgers', 'Bianco Lounge', 'Cha Cha Chai', 'Chopstix Noodle Bar', 'Costa Coffee'] …
same point AND same normalised name: 15 rows

Geometry alone is not identity. A duplicate geometry is a duplicate feature only when the attributes that identify it match too — here the name. to_wkb() compares the binary representation, which is right for points. For polygons it is sensitive to vertex order — two rings describing the same square from different start points are not equal — so normalize() first:

gdf = gdf.assign(geometry=gdf.geometry.normalize())    # canonical vertex order

2. Block by space before comparing text

Comparing every pair of 7,928 rows is 31 million comparisons. A spatial index proposes candidates first. The tempting tool for that is sjoin_nearest — and on a self-join it quietly fails:

nearest = gpd.sjoin_nearest(places[["name", "geometry"]], places[["name", "geometry"]],
                            max_distance=TOLERANCE_M, distance_col="dist").reset_index(names="left_i")
others = nearest[nearest["left_i"] != nearest["index_right"]]
print(f"sjoin_nearest: {len(nearest):,} rows, {len(nearest) - len(others):,} of them a row matched to itself; "
      f"the other {len(others):,} are all at distance {others['dist'].max():.0f} m")

a, b = places.sindex.query(places.geometry.values, predicate="dwithin", distance=TOLERANCE_M)
print(f"dwithin:       {(a < b).sum():,} candidate pairs, at up to {TOLERANCE_M} m")

sjoin_nearest: 15,397 rows, 7,913 of them a row matched to itself; the other 7,484 are all at distance 0 m
dwithin:       27,694 candidate pairs, at up to 50 m

Every row's nearest neighbour in its own layer is itself, at distance zero, so sjoin_nearest only ever reports a second row when it ties at zero — the shared-address points from step 1, which are exactly the pairs you do not want. The real duplicates, a few metres apart, never appear. sindex.query(..., predicate="dwithin") returns every pair within the tolerance, which is what blocking needs.

For polygons, overlap fraction is often a better blocker than distance:

def overlap_fraction(a, b):
    inter = a.intersection(b).area
    return inter / min(a.area, b.area) if min(a.area, b.area) else 0.0

candidates = pairs[[overlap_fraction(g[i], g[j]) > 0.8 for i, j in zip(pairs.a, pairs.b)]]

Two buildings digitised twice overlap almost completely. Two adjacent buildings barely overlap at all. That ratio separates them far more reliably than centroid distance, which is fooled by long thin shapes.

3. Score the text, and choose the right scorer

Four real pairs from the two sources, each within 50 m of its partner:

examples = [("Kirkstall Brewery Taproom And Kitchen", "Kirkstall Brewery Taproom & Kitchen"),
            ("Hare And Hounds", "The Hare and Hounds"),
            ("SALT East Parade", "East Parade Social"),
            ("Greggs", "Greggs PLC")]
print(f"{'':44s} ratio partial token_sort token_set  (normalised token_sort)")
for x, y in examples:
    raw = [fuzz.ratio(x.lower(), y.lower()), fuzz.partial_ratio(x.lower(), y.lower()),
           fuzz.token_sort_ratio(x.lower(), y.lower()), fuzz.token_set_ratio(x.lower(), y.lower())]
    norm = fuzz.token_sort_ratio(normalise_name(x), normalise_name(y))
    print(f"{x[:20]:20s} ↔ {y[:20]:20s}  " + "  ".join(f"{v:5.0f}" for v in raw) + f"   {norm:5.0f}")

                                             ratio partial token_sort token_set  (normalised token_sort)
Kirkstall Brewery Ta ↔ Kirkstall Brewery Ta     94     91     94     97     100
Hare And Hounds      ↔ The Hare and Hounds      88    100     88    100     100
SALT East Parade     ↔ East Parade Social       65     81     88     88      88
Greggs               ↔ Greggs PLC               75    100     75    100     100

token_sort_ratio — right when word order varies.
token_set_ratio — right when one name has extra words, like a legal suffix. Beware: it also scores "High Street" and "High Street West" at 100.
ratio — strictest, right for codes and references.

Normalising first does more than any choice of scorer: removing "the", "and", "&" and "plc" lifts three of the four pairs to 100, and it is the only thing that lets "Greggs PLC" through at all. See how to fuzzy-match place names for the matching side of this. The third pair is the hard case: it passes 85 on the two words it shares, and whether "SALT East Parade" and "East Parade Social" are the same bar is a question for somebody who knows the street, not for a scorer.

Over half the matches within 50 m are more than 10 m apart, so a GPS-style tolerance would miss them; past 50 m chain branches start to join the list. Counts from example 2.

4. Cluster — duplicates come in groups, not pairs

Three records of the same place produce three pairs. Merging pairwise gives the wrong answer; you need the group.

import networkx as nx


def cluster_pairs(pairs, n_rows):
    g = nx.Graph()
    g.add_nodes_from(range(n_rows))
    g.add_edges_from(zip(pairs["a"], pairs["b"]))
    return [sorted(c) for c in nx.connected_components(g) if len(c) > 1]


dupes = find_near_duplicates(places)
clusters = cluster_pairs(dupes, len(places))
print(f"{len(clusters)} clusters covering {sum(map(len, clusters))} rows; "
      f"sizes {pd.Series(map(len, clusters)).value_counts().sort_index().to_dict()}")

661 clusters covering 1323 rows; sizes {2: 660, 3: 1}

Without networkx, a union-find is a dozen lines:

def union_find(pairs, n):
    parent = list(range(n))
    def find(x):
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x
    for a, b in pairs:
        ra, rb = find(a), find(b)
        if ra != rb:
            parent[max(ra, rb)] = min(ra, rb)      # deterministic
    groups = {}
    for i in range(n):
        groups.setdefault(find(i), []).append(i)
    return [v for v in groups.values() if len(v) > 1]

Transitivity is worth watching: A matches B, B matches C, but A and C may be twice the tolerance apart. Connected components will merge all three. Here every cluster but one is a pair, but on a denser layer chains form quickly, so measure the spread of each cluster and look at the widest:

def spread(members):
    pts = places.geometry.values[members]
    return max(p.distance(q) for p in pts for q in pts)


widest = sorted(clusters, key=spread, reverse=True)[:3]
for c in widest:
    print(f"{spread(c):5.1f} m  {places.loc[c, 'source'].tolist()}  {places.loc[c, 'name'].tolist()}")

 49.8 m  ['fsa', 'osm']  ['South Seacroft Friends Lunch Club', 'South Seacroft Friends Lunch Club']
 49.7 m  ['fsa', 'osm']  ['White Cloth Hall', 'White Cloth Hall']
 49.2 m  ['fsa', 'osm']  ['Yorkshire Dance Centre', 'Yorkshire Dance Centre']

5. Decide survivorship explicitly

This is the part people skip, and it is the part that determines whether the result is correct. Three common rules, all defensible, none universal:

Rule	Use when
most complete — fewest nulls	records come from sources of differing detail
most recent — latest date	the world changed and the newest record is right
authoritative source — a `source` ranking	one supplier is definitive and others are supplementary

And the fourth option, which is often the best: merge rather than choose. Here the FSA record is authoritative for the business (it has the registration and the inspection date) and the OSM record contributes its id and its position, which a mapper put on the building rather than on the address:

RANK = {"fsa": 0, "osm": 1}


def merge_group(group):
    """Authoritative source first; fill every empty attribute from the others."""
    ordered = group.sort_values(["source", "name"], key=lambda s: s.map(RANK) if s.name == "source" else s)
    merged = ordered.iloc[0].copy()
    for col in group.columns:
        if col != "geometry" and pd.isna(merged[col]):
            filled = ordered[col].dropna()
            if len(filled):
                merged[col] = filled.iloc[0]
    osm = group[group["source"] == "osm"]
    if len(osm):
        merged["geometry"] = osm.geometry.iloc[0]
    merged["merged_from"] = ",".join(f"{s}:{i}" for s, i in
                                     sorted(zip(group["source"], group["fhrsid"].fillna(group["osm_id"]))))
    merged["merged_count"] = len(group)
    return merged


print(merge_group(places.loc[clusters[0]]).drop("geometry").to_string())

source                                  fsa
fhrsid                              1417349
osm_id                          10212939502
name                                   1915
type                Restaurant/Cafe/Canteen
rating_date             2025-10-23T00:00:00
merged_from     fsa:1417349,osm:10212939502
merged_count                              2

merged_from is the line that makes this reversible. Six months later, "why does this record have that OSM id?" has an answer.

Deduplicating Leeds places across two sources

The data used on this page

Two real sources for the same kind of place, both clipped to the Leeds district (ONS ward boundaries, December 2024) and projected to British National Grid:

FSA Food Hygiene Rating Scheme — every food business Leeds City Council inspects that has coordinates: fhrsid, name, type, rating_date.
OpenStreetMap food and drink POIs (cafés, restaurants, pubs, bars, fast food) from Geofabrik's 1 January 2026 West Yorkshire extract: osm_id, name, type.

import geopandas as gpd
import numpy as np
import pandas as pd
import requests

ONS = "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services"
leeds = gpd.read_file(f"{ONS}/Wards_December_2024_Boundaries_UK_BFC/FeatureServer/0/query"
                      "?where=LAD24NM%3D%27Leeds%27&outFields=WD24CD&outSR=27700&f=geojson")
leeds = leeds.set_crs(27700, allow_override=True).union_all()

fsa, page = [], 1
while True:                                          # the API returns at most 5,000 per page
    batch = requests.get("https://api.ratings.food.gov.uk/Establishments", headers={"x-api-version": "2"},
                         params={"localAuthorityId": 397, "pageSize": 5000, "pageNumber": page},
                         timeout=120).json()["establishments"]
    fsa += batch
    if len(batch) < 5000:
        break
    page += 1
fsa = pd.DataFrame([{"fhrsid": str(e["FHRSID"]), "name": e["BusinessName"], "type": e["BusinessType"],
                     "rating_date": e["RatingDate"], "lon": e["geocode"]["longitude"],
                     "lat": e["geocode"]["latitude"]} for e in fsa]).dropna(subset=["lat"])
fsa = gpd.GeoDataFrame(fsa.drop(columns=["lon", "lat"]), crs=4326,
                       geometry=gpd.points_from_xy(fsa["lon"].astype(float), fsa["lat"].astype(float)))

OSM = "https://download.geofabrik.de/europe/united-kingdom/england/west-yorkshire-260101-free.shp.zip"
with open("west-yorkshire.zip", "wb") as f:
    f.write(requests.get(OSM, timeout=600).content)
osm = gpd.read_file("zip://west-yorkshire.zip!gis_osm_pois_free_1.shp", columns=["osm_id", "name", "fclass"])
osm = osm[osm["fclass"].isin(["cafe", "restaurant", "pub", "bar", "fast_food", "food_court", "nightclub"])
          & osm["name"].notna()].rename(columns={"fclass": "type"})

places = pd.concat([fsa.to_crs(27700).assign(source="fsa"), osm.to_crs(27700).assign(source="osm")],
                   ignore_index=True)
places = gpd.GeoDataFrame(places[places.within(leeds)].reset_index(drop=True), crs=27700)
places = places[["source", "fhrsid", "osm_id", "name", "type", "rating_date", "geometry"]]
print(f"{len(places):,} places: {(places['source'] == 'fsa').sum():,} FSA, {(places['source'] == 'osm').sum():,} OSM")

7,928 places: 5,803 FSA, 2,125 OSM

Example 1: the complete deduplication

def deduplicate(gdf, *, tol=TOLERANCE_M, score=NAME_SCORE):
    gdf = gdf.reset_index(drop=True)
    report = {"in": len(gdf)}

    # exact: same point and same normalised name
    exact = pd.Series(list(zip(gdf.geometry.to_wkb(), gdf["name"].map(normalise_name)))).duplicated()
    report["exact_removed"] = int(exact.sum())
    gdf = gdf.loc[~exact.values].reset_index(drop=True)

    # candidates, text confirmation, clusters
    matched = find_near_duplicates(gdf, tol=tol, score=score)
    report["matched_pairs"] = len(matched)
    clusters = cluster_pairs(matched, len(gdf))
    report["clusters"] = len(clusters)

    # merge
    in_cluster = [i for c in clusters for i in c]
    merged = gpd.GeoDataFrame([merge_group(gdf.loc[c]) for c in clusters], crs=gdf.crs)
    out = pd.concat([gdf.drop(index=in_cluster), merged], ignore_index=True)

    report["out"] = len(out)
    report["rows_merged_away"] = report["in"] - report["exact_removed"] - report["out"]
    report["with_both_ids"] = int((out["fhrsid"].notna() & out["osm_id"].notna()).sum())
    return gpd.GeoDataFrame(out, crs=gdf.crs), report, matched, clusters


clean, report, pairs, clusters = deduplicate(places)
for k, v in report.items():
    print(f"{k:18s} {v:>6,}")

in                  7,913
exact_removed           0
matched_pairs         663
clusters              661
out                 7,251
rows_merged_away      662
with_both_ids         650

Example 2: tuning the thresholds with evidence, not intuition

The name threshold first. Reading the weakest matches at each level is how you pick one:

wide = find_near_duplicates(places, score=0)
names = places["name"].values
for t in (70, 80, 85, 90, 100):
    hits = wide[wide["score"] >= t]
    print(f"score ≥ {t:3}: {len(hits):>4} pairs; weakest: " + "; ".join(
        f"{names[r.a]!r} ↔ {names[r.b]!r}" for r in hits.nsmallest(2, "score").itertuples()))

score ≥  70:  809 pairs; weakest: 'Middleton Chinese' ↔ 'Boots The Chemist (Middleton)'; 'New Inn Eccup' ↔ 'The New Inn'
score ≥  80:  699 pairs; weakest: 'Bootlegger Bars' ↔ 'Bootlegger'; 'Brooklyn Bar' ↔ 'Brooklyn'
score ≥  85:  663 pairs; weakest: 'Morrisons (Hunslet)' ↔ 'Morrisons Daily Hunslet'; 'Otley Golf Club' ↔ '1906 @ Otley Golf Club'
score ≥  90:  634 pairs; weakest: 'Cafe Oasis' ↔ 'Café Oasis'; 'Caffe Nero' ↔ 'Caffè Nero'
score ≥ 100:  573 pairs; weakest: '1915' ↔ '1915'; '92 Degrees' ↔ '92 Degrees'

Then the distance. Widen the search to 200 m and watch what kind of name starts matching:

far = find_near_duplicates(places, tol=200)
far["name"] = places["name"].values[far["a"]]
far["band"] = pd.cut(far["dist"], [0, 10, 25, 50, 100, 200], right=False)
print(far.groupby("band", observed=True).size().to_string())
print("most repeated names beyond 50 m:",
      far.loc[far["dist"] >= 50, "name"].value_counts().head(5).to_dict())

band
[0, 10)       259
[10, 25)      202
[25, 50)      202
[50, 100)     191
[100, 200)    123
most repeated names beyond 50 m: {'Greggs': 12, 'Caffe Nero': 8, 'Subway': 7, 'Starbucks': 4, 'Caffè Nero': 4}

Past 50 m the list fills with chains — Greggs, Caffè Nero, Subway — whose branches really are a street apart. Plenty of genuine duplicates also sit that far apart, because an address point can be the far side of a large building from its entrance. No single number separates them. Counting how many matches land on a chain name at each setting shows the trade directly:

normalised = places["name"].map(normalise_name)
uses = normalised.map(normalised.value_counts()).values
for tol, score in [(10, 92), (50, 85), (200, 80)]:
    hits = find_near_duplicates(places, tol=tol, score=score)
    print(f"{tol:>3} m / score {score}: {len(hits):>5} pairs, "
          f"{(uses[hits['a']] >= 5).sum():>3} on a name used 5+ times in Leeds")

 10 m / score 92:   234 pairs,  16 on a name used 5+ times in Leeds
 50 m / score 85:   663 pairs,  49 on a name used 5+ times in Leeds
200 m / score 80:  1038 pairs, 110 on a name used 5+ times in Leeds

The tight setting finds well under half the matches the middle one does; the loose one adds several hundred more, and more than twice as many of them are on chain names. What does work is a tolerance that depends on the name: generous for a name that occurs once in the layer, tight for one that occurs twenty times.

Example 3: reviewing before committing

def export_for_review(gdf, clusters, path):
    """One row per cluster member, grouped, so a human can scan it in QGIS."""
    rows = []
    for cid, members in enumerate(clusters):
        sub = gdf.loc[members]
        for r in sub.itertuples():
            rows.append({"cluster": cid, "source": r.source, "name": r.name,
                         "members": len(members), "spread_m": round(spread(members), 1),
                         "geometry": r.geometry})
    out = gpd.GeoDataFrame(rows, crs=gdf.crs).sort_values(["spread_m", "cluster"], ascending=[False, True])
    out.to_file(path, driver="GPKG")
    return out


review = export_for_review(places, clusters, "review_duplicates.gpkg")
print(review.head(6)[["cluster", "source", "name", "members", "spread_m"]].to_string(index=False))

 cluster source                              name  members  spread_m
     313    fsa South Seacroft Friends Lunch Club        2      49.8
     313    osm South Seacroft Friends Lunch Club        2      49.8
     406    fsa                  White Cloth Hall        2      49.7
     406    osm                  White Cloth Hall        2      49.7
      73    fsa                   Cucina Pizzeria        2      49.2
      73    osm                      Cucina Pizza        2      49.2

Sort by spread_m descending and the doubtful clusters float to the top — those are the ones where transitivity may have chained together things that are not the same.

A record linkage problem with no error-free threshold

Loosening finds more real duplicates and more chain branches at the same time; there is no setting that gets one without the other.

Near-duplicate detection is a record linkage problem, and it inherits record linkage's central fact: there is no threshold that produces zero errors. Tighten it and real duplicates survive; loosen it and distinct things merge. The only question is which error is more expensive for the use you have in mind.

That asymmetry usually has a clear answer. For a mailing list, a false merge means one letter instead of two — annoying. For an asset register, a false merge means a bridge disappears from the maintenance schedule — serious. When a false merge is expensive, set the threshold high and flag the rest for review rather than tuning until the count looks right.

The spatial dimension is what makes this tractable compared with plain text deduplication. Two identically named shops in different cities are not duplicates, and a purely textual matcher cannot tell. Blocking on proximity first uses the spatial index to reduce a quadratic problem to a nearly linear one, and simultaneously removes the largest class of false positives for free. It is both the performance fix and the accuracy fix.

Clustering matters more than it appears. Duplicates arrive in groups — the same hospital in a survey, an address file and an OSM extract — and pairwise merging processes A+B, then discovers C matches a row that no longer exists. Building the graph first and merging each connected component once is the only approach that terminates with the right answer.

Finally, survivorship is where the information is lost. Every other step is reversible in principle; choosing one record and discarding two is not, unless you record what was discarded. That is what merged_from and merged_count are for, and why writing the merged-away rows to a side file costs nothing and answers every later question.

Self-matches, transitive over-merging and token_set_ratio

sjoin_nearest against the same layer finds each row itself, at distance 0, and nothing else unless points coincide exactly. Use sindex.query(..., predicate="dwithin") for blocking, in a projected CRS so the distance is in metres.
to_wkb().duplicated() is vertex-order sensitive. normalize() first for a canonical order.
Transitivity can over-merge. Cap cluster diameter or use a stricter threshold when chains appear.
token_set_ratio scores "High Street" and "High Street West" at 100. It is the most permissive scorer and needs the highest threshold.
rapidfuzz implements the same scorers as the older fuzzywuzzy, in C++ and under the MIT licence rather than the GPL.
Merging polygons changes area. Decide whether the survivor keeps its own geometry or the union of the group — they are different answers.
Deduplication is not idempotent by default. Run it twice and clusters can shift if tie-breaks depend on row order; sort explicitly.
Keep the removed rows. A rejected_duplicates.gpkg beside the output costs nothing.

Internal links

How to find and remove duplicate geometries in GeoPandas — the exact-match case
How to fuzzy-match place names when joining spatial data — the same scoring, for joining rather than merging
Repair, reject or flag? Choosing what cleaning should do — survivorship as a cleaning decision
Spatial indexes explained: R-trees and why spatial joins are fast — why blocking works
How to find the nearest point in GeoPandas — sjoin_nearest in general
GeoPandas spatial join returns duplicate rows — duplicates created by a join rather than found in data
How to clean and normalise attribute columns in a GeoDataFrame — normalising before scoring
The Python GIS data cleaning checklist — where this step belongs

FAQ

What distance tolerance should I use?

It depends on how the two sources placed their points, so measure it. On this page, of the FSA–OSM matches within 50 m, 259 were within 10 m and 202 were between 25 and 50 m — address points and mapped entrances really are that far apart. Widen the search until the new matches start being chain branches rather than the same place, and stop just before.

Which fuzzy scorer is best for place names?

token_sort_ratio at around 85. It handles word-order differences without being as permissive as token_set_ratio, which scores any name containing another at 100.

Why cluster instead of merging pairs?

Duplicates come in groups. Pairwise merging processes A+B and then finds C matching a row that no longer exists. Connected components handle the group in one pass.

How do I stop two different shops on one street merging?

Tighten the distance tolerance, use overlap fraction rather than centroid distance for polygons, and require a stricter name score. Then review the weakest matches.

Which record should survive?

Whichever rule you can defend: most complete, most recent, or most authoritative. Better still, merge — take the best geometry and fill each attribute from the first record that has it.

How do I make it reproducible?

Sort before every tie-break and never rely on file order. Deduplication that depends on row order gives different answers on different machines.

Should I delete the duplicates?

Merge them and record what was merged, with a merged_from column and a side file of the removed rows. Deletion without a record cannot be reviewed.