Address Matching Explained: Why Exact String Equality Fails

Problem statement

You have two files that both contain addresses — a customer list and a property register, this month's export and last month's — and you need to know which rows describe the same place.

The obvious implementation is a join on the address column. It fails, and it fails much harder than people expect.

A controlled measurement shows how badly. Take the 236,014 populated places in GeoNames' cities500 file as the reference, draw 5,000 of them, and apply to each exactly one everyday variation — uppercase it, strip its accents, swap Saint for St, drop one letter, turn a hyphen into a space, or add a stray space. No person would call any of those a different place.

Exact string equality matched 1.1%, and every one of those hits was the wrong place: a dropped letter that happened to spell another town. Example 4 is the experiment, so it can be re-run and changed.

Quick answer

Match in three tiers, and only escalate the rows that failed the tier above:

def match(queries, reference):
    """Tier 1: normalised exact. Tier 2: blocked candidates. Tier 3: fuzzy."""
    index = {}
    for r in reference:
        index.setdefault(normalise(r.name), []).append(r)

    matched, residue = {}, []
    for q in queries:
        hits = index.get(normalise(q.text))
        if hits:
            matched[q.id] = hits            # may be more than one — see below
        else:
            residue.append(q)

    for q in residue:                       # only these reach the expensive path
        best = fuzzy_best(q, candidates_for(q, index))
        if best and best.score >= 0.85:
            matched[q.id] = [best.record]
    return matched, [q for q in residue if q.id not in matched]

The measured effect of each tier on the same 5,000 queries, from Example 4:

verbatim                 1.1%   (all of them the wrong place)
+ lowercase             31.2%
+ strip accents         35.7%
+ collapse punctuation  71.9%
+ expand abbreviations  71.9%
+ fuzzy on the residue  184 of 200 right, at about 70 ms per lookup

Normalise and know the costs, break ties

1. Normalise, and know what each step buys

The ladder above is worth reading as a set of priorities. Case folding added 30 points. Collapsing punctuation and whitespace added 36.2 — the single largest step, and the one nobody demonstrates in a blog post, because a trailing space is invisible in the data.

Abbreviation expansion added nothing measurable. It is the step people build first and the one that matters least here: few place names contain an abbreviation to vary, and on real address files the visible ones have usually been fixed by hand already.

2. Understand what normalisation costs you

Normalisation merges, and merging creates ambiguity. Place names repeat before any normalising — 17,121 of the verbatim names already belong to two or more places — and after it the 236,014 records form 198,362 keys, of which 17,958, 9.1%, point at more than one distinct place.

So a normalised match is not automatically a correct match. Measured directly: 50 of the 5,000 queries landed on a key that does not include the place they came from, about 1.4% of the matches.

That is the trade: normalisation converts a 99% miss rate into about a 1% wrong-match rate. It is an excellent trade, and it is not free, which is why the ambiguous keys need a second field to break the tie.

3. Break ties with a second field, not a better string algorithm

When a normalised key matches several records, no amount of string cleverness resolves it — the strings are identical. What resolves it is more information:

• Postcode — the strongest tiebreaker for addresses, and independently sourced.
• Region or country — resolves most place-name collisions; 6,655 normalised names in cities500 occur in more than one country.
• Population or prominence — resolves "which Springfield" the way a search engine would, which is a guess, but a documented one.
• Distance to a known point — if you already have an approximate location, the nearest candidate is almost always right.

4. Block before you compare

Fuzzy matching every query against every reference record is a Cartesian product: 5,000 × 236,014 is 1.2 billion comparisons. Blocking restricts the comparison to candidates that share something cheap and discriminating:

• the postcode, or its outward part
• the first three characters of the normalised name
• a phonetic key such as Soundex or Metaphone
• the country plus the length of the name, bucketed

A good block cuts the candidate set by three or four orders of magnitude while keeping essentially all true matches. A bad block — one on a field that is itself misspelled — silently discards them.

5. Escalate only the residue

After normalisation, 1,403 of the 5,000 queries — 28.1%, nearly all of them dropped letters — remained unmatched. Fuzzy matching against every key found the right place for 184 of a sample of 200 of them, and a wrong one for the other 16, at about 70 ms per lookup against 0.03 microseconds for the dictionary lookup itself.

That is a factor of about two million. Fuzzy matching the whole file is not a more thorough approach; it is the same result, computed the expensive way on rows that did not need it.

6. Record how each row matched

A match with no explanation cannot be audited. Store the tier (exact, normalised, fuzzy), the score, the number of candidates and the tiebreaker used. When somebody asks why a customer was merged into the wrong property, that record is the answer.

Blocked fuzzy matching with an audit

The data used on this page

GeoNames cities500: every populated place with at least 500 people, 236,014 of them when this page was run. GeoNames is edited continuously, so the counts drift a little between downloads.

import io
import os
import zipfile

import pandas as pd
import requests

COLS = ["geonameid", "name", "asciiname", "alternatenames", "lat", "lon", "fclass", "fcode", "country",
        "cc2", "admin1", "admin2", "admin3", "admin4", "population", "elevation", "dem", "timezone", "modified"]
if not os.path.exists("cities500.parquet"):
    # GeoNames cities500 as saved on 8 October 2026 (rebuilt nightly); live: https://download.geonames.org/export/dump/cities500.zip
    z = zipfile.ZipFile(io.BytesIO(requests.get("https://www.spatialworkflow.io/data/geonames-cities500-2026-10-08.zip",
                                                timeout=600).content))
    (pd.read_csv(z.open("cities500.txt"), sep="\t", names=COLS, dtype={"admin1": str}, quoting=3,
                 keep_default_na=False, na_values=[""])
     [["geonameid", "name", "country", "admin1", "population"]].to_parquet("cities500.parquet"))
places = pd.read_parquet("cities500.parquet")
print(f"{len(places):,} populated places, GeoNames cities500; "
      f"{places.name.str.contains(r'[^\x00-\x7F]').mean():.1%} of the names are not plain ASCII")

236,014 populated places, GeoNames cities500; 20.4% of the names are not plain ASCII

Example 1 — blocked fuzzy matching

from collections import defaultdict
import difflib


def build_blocks(reference, block_key):
    blocks = defaultdict(list)
    for record in reference:
        blocks[block_key(record)].append(record)
    return blocks


def postcode_block(record):
    """Outward code: 'NW1 8QP' -> 'NW1'. Cheap, discriminating, usually present."""
    pc = (record.postcode or "").upper().replace(" ", "")
    return pc[:-3] if len(pc) > 3 else pc


def fuzzy_match(query, blocks, block_key, cutoff=0.85, max_candidates=200):
    candidates = blocks.get(block_key(query), [])
    if not candidates:
        return None, 0.0, 0
    if len(candidates) > max_candidates:      # a block this big is not a block
        return None, 0.0, len(candidates)

    qn = normalise(query.text)
    best, best_score = None, 0.0
    for candidate in candidates:
        score = difflib.SequenceMatcher(None, qn, normalise(candidate.name)).ratio()
        if score > best_score:
            best, best_score = candidate, score
    return (best, best_score, len(candidates)) if best_score >= cutoff \
        else (None, best_score, len(candidates))

The max_candidates guard is a bug-catcher. A block holding thousands of records means the blocking key is not discriminating — usually because it is empty for those rows, and every empty value collided into one bucket.

Example 2 — the tiered matcher with an audit trail

from dataclasses import dataclass


@dataclass
class MatchResult:
    query_id: str
    record_id: str | None
    tier: str              # exact | normalised | fuzzy | ambiguous | none
    score: float
    candidates: int
    tiebreaker: str | None = None


def match_all(queries, reference, block_key=postcode_block, cutoff=0.85):
    exact = {r.name: r for r in reference}
    norm_index = defaultdict(list)
    for r in reference:
        norm_index[normalise(r.name)].append(r)
    blocks = build_blocks(reference, block_key)

    results = []
    for q in queries:
        if q.text in exact:
            results.append(MatchResult(q.id, exact[q.text].id, "exact", 1.0, 1))
            continue

        hits = norm_index.get(normalise(q.text), [])
        if len(hits) == 1:
            results.append(MatchResult(q.id, hits[0].id, "normalised", 1.0, 1))
            continue
        if len(hits) > 1:
            picked, why = break_tie(q, hits)
            results.append(MatchResult(q.id, picked.id if picked else None,
                                       "ambiguous", 1.0, len(hits), why))
            continue

        best, score, n = fuzzy_match(q, blocks, block_key, cutoff)
        results.append(MatchResult(q.id, best.id if best else None,
                                   "fuzzy" if best else "none", round(score, 3), n))
    return results


def break_tie(query, candidates):
    if getattr(query, "postcode", None):
        same = [c for c in candidates if c.postcode == query.postcode]
        if len(same) == 1:
            return same[0], "postcode"
    if getattr(query, "country", None):
        same = [c for c in candidates if c.country == query.country]
        if len(same) == 1:
            return same[0], "country"
    ranked = sorted(candidates, key=lambda c: -(c.population or 0))
    if ranked and (ranked[0].population or 0) > 10 * (ranked[1].population or 1):
        return ranked[0], "population (10x)"
    return None, "unresolved"

break_tie returning None is a feature. An unresolved ambiguity should be reported, not decided by whichever record the index happened to hold first.

Example 3 — measuring your own match ladder

def ladder(queries, reference_names, strategies):
    """What does each normalisation step buy on YOUR data?"""
    for label, fn in strategies.items():
        index = {}
        for name in reference_names:
            index.setdefault(fn(name), set()).add(name)
        hits = sum(1 for q in queries if fn(q) in index)
        ambiguous = sum(1 for v in index.values() if len(v) > 1)
        print(f"{label:22s} {100 * hits / len(queries):5.1f}%   "
              f"keys {len(index):7,}  ambiguous {ambiguous:6,}")

Run this before adopting anyone's recipe. The ordering of the steps is stable across datasets; the size of each step is not, and knowing which one carries your data tells you where to spend effort.

Example 4 — the experiment behind the numbers on this page

The reference is GeoNames' cities500; the queries are 5,000 of its own names, each given one variation that changes the string. The mix of variations decides the size of each step, so change perturb to match your own data before trusting the ladder.

import difflib
import hashlib
import random
import re
import time
import unicodedata


def fold_accents(s):
    return "".join(c for c in unicodedata.normalize("NFKD", s) if not unicodedata.combining(c))


PLACE_ABBREV = {"st": "saint", "ste": "sainte", "mt": "mount", "ft": "fort", "pt": "point"}
STEPS = {
    "verbatim":               lambda s: s,
    "+ lowercase":            lambda s: s.casefold(),
    "+ strip accents":        lambda s: fold_accents(s.casefold()),
    "+ collapse punctuation": lambda s: " ".join(re.sub(r"[^\w\s]", " ", fold_accents(s.casefold())).split()),
    "+ expand abbreviations": lambda s: " ".join(PLACE_ABBREV.get(t, t) for t in
                                                 re.sub(r"[^\w\s]", " ", fold_accents(s.casefold())).split()),
}
normalise = STEPS["+ expand abbreviations"]

SWAPS = [("Saint ", "St "), ("St. ", "Saint "), ("Mount ", "Mt "), ("Fort ", "Ft "), ("Sainte ", "Ste ")]


def perturb(name, rng):
    """One everyday variation, chosen among those that apply to this name."""
    options = [lambda s: s + " "]
    if name.upper() != name:
        options.append(lambda s: s.upper())
    if " " in name:
        options.append(lambda s: s.replace(" ", "  ", 1))
    if fold_accents(name) != name:
        options.append(fold_accents)
    if "-" in name:
        options.append(lambda s: s.replace("-", " "))
    options += [lambda s, a=a, b=b: s.replace(a, b, 1) for a, b in SWAPS if a in name]
    if len(name) >= 5:
        i = rng.randrange(1, len(name) - 1)
        options.append(lambda s, i=i: s[:i] + s[i + 1:])
    return rng.choice(options)(name)                  # every option changes the string


rng = random.Random(42)
# a fixed 5,000: the places whose ids hash lowest, so GeoNames' daily edits barely move the sample
sample = places.loc[places.geonameid.map(lambda i: hashlib.sha256(str(i).encode()).hexdigest())
                    .sort_values().index[:5000]]
queries = [(gid, perturb(name, rng)) for gid, name in zip(sample.geonameid, sample.name)]

for label, fn in STEPS.items():
    index = {}
    for gid, name in zip(places.geonameid, places.name):
        index.setdefault(fn(name), set()).add(gid)
    keys = [fn(q) for _, q in queries]
    hit = sum(k in index for k in keys)
    wrong = sum(k in index and gid not in index[k] for (gid, _), k in zip(queries, keys))
    shared = sum(len(v) > 1 for v in index.values())
    print(f"{label:24s} {hit:5,} {100 * hit / len(queries):5.1f}%   wrong place {wrong:3}   "
          f"keys {len(index):7,}  shared by 2+ places {shared:6,}")

residue = [(gid, q) for gid, q in queries if normalise(q) not in index]
print(f"\nafter normalisation {len(residue):,} of {len(queries):,} unmatched")
keys = list(index)
trial = random.Random(1).sample(residue, 200)
start = time.perf_counter()
found = [difflib.get_close_matches(normalise(q), keys, n=1, cutoff=0.85) for _, q in trial]
per_fuzzy = (time.perf_counter() - start) / len(trial)
right = sum(bool(m) and gid in index[m[0]] for (gid, _), m in zip(trial, found))
qk = [normalise(q) for _, q in trial]
start = time.perf_counter()
for _ in range(1000):
    for k in qk:
        k in index
per_hash = (time.perf_counter() - start) / (1000 * len(qk))
print(f"fuzzy on 200 of them: {sum(map(bool, found))} found a candidate, {right} the right place")
countries = places.assign(k=places.name.map(normalise)).groupby("k").country.nunique()
print(f"normalised names in more than one country: {(countries > 1).sum():,}")

verbatim                    54   1.1%   wrong place  54   keys 200,073  shared by 2+ places 17,121
+ lowercase              1,562  31.2%   wrong place  55   keys 200,052  shared by 2+ places 17,133
+ strip accents          1,785  35.7%   wrong place  51   keys 198,495  shared by 2+ places 17,918
+ collapse punctuation   3,596  71.9%   wrong place  50   keys 198,402  shared by 2+ places 17,954
+ expand abbreviations   3,597  71.9%   wrong place  50   keys 198,362  shared by 2+ places 17,958

after normalisation 1,403 of 5,000 unmatched
fuzzy on 200 of them: 200 found a candidate, 184 the right place
normalised names in more than one country: 6,655

The fuzzy step's cost against an exact lookup, on this machine:

print(f"{per_fuzzy * 1000:.0f} ms a fuzzy lookup against {per_hash * 1e6:.2f} µs for the dictionary")

73 ms a fuzzy lookup against 0.03 µs for the dictionary

Example 5 — what a block costs

import numpy as np

keyed = places.assign(key=places.name.map(normalise))
keyed["prefix"] = keyed.key.str[:3]
truth = keyed.set_index("geonameid")
blocks = {
    "every record":              (lambda q, c: 0, {0: len(keyed)}, lambda g: 0),
    "same country":              (lambda q, c: c, keyed.groupby("country").size(), lambda g: truth.country[g]),
    "country + first 3 letters": (lambda q, c: (c, normalise(q)[:3]), keyed.groupby(["country", "prefix"]).size(),
                                  lambda g: (truth.country[g], truth.prefix[g])),
}
for label, (block_of, sizes, true_block) in blocks.items():
    n = [sizes.get(block_of(q, truth.country[g]), 0) for g, q in queries]
    kept = np.mean([block_of(q, truth.country[g]) == true_block(g) for g, q in queries])
    print(f"{label:26} {np.mean(n):9,.0f} candidates a query, true match in the block {kept:.1%}")

every record                 236,014 candidates a query, true match in the block 100.0%
same country                   8,838 candidates a query, true match in the block 100.0%
country + first 3 letters         96 candidates a query, true match in the block 91.5%

The country block is free: it cuts the candidates by a factor of 27 and keeps every true match, because the country was never perturbed. The prefix block cuts them by another 92 and loses 8.5% of the true matches — the queries whose dropped letter or stray character fell in the first three. That is the warning in step 4, measured.

Why invisible differences break exact matches

Why exact matching fails on differences nobody would notice

The intuition that string matching "mostly works" comes from clean examples. Real address and place data varies along six axes at once — case, accents, punctuation, whitespace, abbreviation and typos — and a single difference on any axis is a complete miss.

The measured perturbations were each individually trivial. That is the point: no human would call any of them a different address, and string equality calls all of them different.

Why collapsing punctuation and whitespace carried the largest gain

Punctuation and whitespace variation is invisible on screen. A trailing space, a double space, a hyphen instead of a space, a comma instead of a hyphen — nobody sees these in a spreadsheet, so nobody fixes them by hand, so they accumulate.

Case and accents are half-visible: people notice LONDON and sometimes fix it. Abbreviations are fully visible, so in real files they are usually already consistent by the time the data reaches you.

Why fuzzy matching is a last resort rather than a strategy

Fuzzy matching is genuinely effective: 184 of 200 right at a 0.85 cutoff. It is also millions of times slower per lookup than a dictionary hit, and it introduces a threshold that has to be tuned, which means it introduces false matches — 16 of those 200.

The tiered design puts it where it earns its cost — on the 28.1% residue — and keeps the cheap path cheap. It also makes the threshold a decision about a small, inspectable set of rows rather than about the whole file.

Why the tiebreaker is a data problem, not an algorithm problem

When two records normalise to the same key, the strings carry no remaining information. Every additional comparison is arithmetic on identical inputs.

The only thing that can resolve it is a field that differs: postcode, region, population, an approximate coordinate. This is why address matching is really record linkage, and why the quality of the match depends far more on which fields you have than on which string algorithm you chose.

Misspelled blocks and phonetic keys

• Blocking on a misspelled field silently loses true matches. Block on the most reliable field you have, and consider two passes with different blocks.
• Phonetic keys help for names and hurt for codes. Soundex on a postcode is noise.
• A 0.85 cutoff is a starting point, not a constant. Calibrate on labelled pairs; the right value depends on name length.
• Short names match everything. Guard against fuzzy-matching strings under about five characters.
• Transliteration is not normalisation. München and Munich are the same city and no accent fold connects them; that needs an alternate-names table.
• Record the tier. An audit that cannot distinguish an exact match from a 0.86 fuzzy match is not an audit.
• One-to-many is the normal outcome, not an error. 9.1% of normalised keys had more than one place behind them.
• Never match on the raw string and the normalised string in the same join — you will double-count.

Internal links

• How to parse and normalise addresses in Python — the normaliser used here
• How to match addresses against a reference file — this design, implemented end to end
• How to deduplicate an address list in Python — the same machinery pointed at one file
• How to fuzzy match place names in Python — the fuzzy tier in more depth
• The address data model: why an address is not a string — the fields that break ties
• Match quality explained: reading a geocoder's confidence score — the same ambiguity, inside a geocoder
• Fixing a geocoder that matches the wrong town — what an unresolved tie looks like downstream
• How to merge near-duplicate features — the geometric version of this problem

FAQ

Why does joining on the address column miss so many rows?

Because the same name is written many ways. In the controlled test on this page, exact string equality matched 1.1% of place names with one everyday variation; normalisation took it to 71.9%.

Which normalisation step matters most?

Collapsing punctuation and whitespace, which added 36.2 points in the measured ladder, followed by case folding at 30. Abbreviation expansion added nothing measurable.

Does normalisation cause wrong matches?

A few. 9.1% of normalised keys covered more than one distinct place, and about 1% of matches landed on a place other than the one the query came from. That is the price of a 71-point recall gain.

When should I use fuzzy matching?

Only on the rows normalisation could not place — 28.1% in the measured run. It found the right place for 184 of 200 of those, at about 70 ms per lookup against 0.03 microseconds for a dictionary lookup.

How do I choose a blocking key?

Something cheap, discriminating and reliably present: the outward postcode, the first characters of the normalised name, or the country. Check the block-size distribution — a huge block means the key is missing for those rows.

What do I do with ambiguous matches?

Break the tie with a second field — postcode, region, population — and record which one you used. If nothing resolves it, report it rather than picking the first candidate.