Address Matching Explained: Why Exact String Equality Fails
Problem statement
You have two files that both contain addresses — a customer list and a property register, this month's export and last month's — and you need to know which rows describe the same place.
The obvious implementation is a join on the address column. It fails, and it fails much harder than people expect.
In a controlled measurement, 5,000 realistically varied queries were matched against a reference set of 238,483 places. Every query was a real name with exactly one everyday perturbation applied: uppercased, accents stripped, an abbreviation expanded or contracted, one character dropped, a hyphen turned into a space, or stray whitespace added.
Exact string equality matched 29.1%. Seven rows in ten did not join, and none of them were wrong — they were the same places, written differently.
Quick answer
Match in three tiers, and only escalate the rows that failed the tier above:
def match(queries, reference):
"""Tier 1: normalised exact. Tier 2: blocked candidates. Tier 3: fuzzy."""
index = {}
for r in reference:
index.setdefault(normalise(r.name), []).append(r)
matched, residue = {}, []
for q in queries:
hits = index.get(normalise(q.text))
if hits:
matched[q.id] = hits # may be more than one — see below
else:
residue.append(q)
for q in residue: # only these reach the expensive path
best = fuzzy_best(q, candidates_for(q, index))
if best and best.score >= 0.85:
matched[q.id] = [best.record]
return matched, [q for q in residue if q.id not in matched]The measured effect of each tier on the same 5,000 queries:
verbatim 29.1%
+ lowercase 54.7%
+ strip accents 58.1%
+ collapse punctuation 85.4%
+ expand abbreviations 85.5%
+ fuzzy on the residue ~88% at 134 ms per lookup
Step-by-step solution
1. Normalise, and know what each step buys
The ladder above is worth reading as a set of priorities. Case folding alone nearly doubled the match rate. Collapsing punctuation and whitespace added another 27 points — the single largest step, and the one nobody demonstrates in a blog post, because it is invisible in the data.
Abbreviation expansion added 0.1 points. It is the step people build first and the one that matters least, because abbreviations are visible: somebody has already fixed most of them by hand.
2. Understand what normalisation costs you
Normalisation merges, and merging creates ambiguity. In the same reference set, the 238,483 records collapsed to 190,028 normalised keys, of which 20,765 — 10.9% — pointed at more than one distinct place.
So a normalised match is not automatically a correct match. Measured directly: 38 of the 5,000 matched queries landed on a different place with the same normalised name, a 0.8% wrong-target rate.
That is the trade: normalisation converts a 71% miss rate into a 0.8% wrong-match rate. It is an excellent trade, and it is not free, which is why the ambiguous keys need a second field to break the tie.
3. Break ties with a second field, not a better string algorithm
When a normalised key matches several records, no amount of string cleverness resolves it — the strings are identical. What resolves it is more information:
- Postcode — the strongest tiebreaker for addresses, and independently sourced.
- Region or country — resolves the majority of place-name collisions;
132,528place names occur in more than one country. - Population or prominence — resolves "which Springfield" the way a search engine would, which is a guess, but a documented one.
- Distance to a known point — if you already have an approximate location, the nearest candidate is almost always right.
4. Block before you compare
Fuzzy matching every query against every reference record is a Cartesian product: 5,000 × 238,483 is 1.2 billion comparisons. Blocking restricts the comparison to candidates that share something cheap and discriminating:
- the postcode, or its outward part
- the first three characters of the normalised name
- a phonetic key such as Soundex or Metaphone
- the country plus the length of the name, bucketed
A good block cuts the candidate set by three or four orders of magnitude while keeping essentially all true matches. A bad block — one on a field that is itself misspelled — silently discards them.
5. Escalate only the residue
After normalisation, 724 of the 5,000 queries — 14.5% — remained unmatched. Fuzzy matching recovered 188 of a sample of 200 of them, at 134 ms per lookup against 3.3 microseconds for a normalised dictionary hit.
That is a factor of forty thousand. Fuzzy matching the whole file is not a more thorough approach; it is the same result, computed the expensive way on rows that did not need it.
6. Record how each row matched
A match with no explanation cannot be audited. Store the tier (exact, normalised, fuzzy), the score, the number of candidates and the tiebreaker used. When somebody asks why a customer was merged into the wrong property, that record is the answer.
Code examples
Example 1 — blocked fuzzy matching
from collections import defaultdict
import difflib
def build_blocks(reference, block_key):
blocks = defaultdict(list)
for record in reference:
blocks[block_key(record)].append(record)
return blocks
def postcode_block(record):
"""Outward code: 'NW1 8QP' -> 'NW1'. Cheap, discriminating, usually present."""
pc = (record.postcode or "").upper().replace(" ", "")
return pc[:-3] if len(pc) > 3 else pc
def fuzzy_match(query, blocks, block_key, cutoff=0.85, max_candidates=200):
candidates = blocks.get(block_key(query), [])
if not candidates:
return None, 0.0, 0
if len(candidates) > max_candidates: # a block this big is not a block
return None, 0.0, len(candidates)
qn = normalise(query.text)
best, best_score = None, 0.0
for candidate in candidates:
score = difflib.SequenceMatcher(None, qn, normalise(candidate.name)).ratio()
if score > best_score:
best, best_score = candidate, score
return (best, best_score, len(candidates)) if best_score >= cutoff \
else (None, best_score, len(candidates))The max_candidates guard is a bug-catcher. A block holding thousands of records means the blocking key is not discriminating — usually because it is empty for those rows, and every empty value collided into one bucket.
Example 2 — the tiered matcher with an audit trail
from dataclasses import dataclass
@dataclass
class MatchResult:
query_id: str
record_id: str | None
tier: str # exact | normalised | fuzzy | ambiguous | none
score: float
candidates: int
tiebreaker: str | None = None
def match_all(queries, reference, block_key=postcode_block, cutoff=0.85):
exact = {r.name: r for r in reference}
norm_index = defaultdict(list)
for r in reference:
norm_index[normalise(r.name)].append(r)
blocks = build_blocks(reference, block_key)
results = []
for q in queries:
if q.text in exact:
results.append(MatchResult(q.id, exact[q.text].id, "exact", 1.0, 1))
continue
hits = norm_index.get(normalise(q.text), [])
if len(hits) == 1:
results.append(MatchResult(q.id, hits[0].id, "normalised", 1.0, 1))
continue
if len(hits) > 1:
picked, why = break_tie(q, hits)
results.append(MatchResult(q.id, picked.id if picked else None,
"ambiguous", 1.0, len(hits), why))
continue
best, score, n = fuzzy_match(q, blocks, block_key, cutoff)
results.append(MatchResult(q.id, best.id if best else None,
"fuzzy" if best else "none", round(score, 3), n))
return results
def break_tie(query, candidates):
if getattr(query, "postcode", None):
same = [c for c in candidates if c.postcode == query.postcode]
if len(same) == 1:
return same[0], "postcode"
if getattr(query, "country", None):
same = [c for c in candidates if c.country == query.country]
if len(same) == 1:
return same[0], "country"
ranked = sorted(candidates, key=lambda c: -(c.population or 0))
if ranked and (ranked[0].population or 0) > 10 * (ranked[1].population or 1):
return ranked[0], "population (10x)"
return None, "unresolved"break_tie returning None is a feature. An unresolved ambiguity should be reported, not decided by whichever record the index happened to hold first.
Example 3 — measuring your own match ladder
def ladder(queries, reference_names, strategies):
"""What does each normalisation step buy on YOUR data?"""
for label, fn in strategies.items():
index = {}
for name in reference_names:
index.setdefault(fn(name), set()).add(name)
hits = sum(1 for q in queries if fn(q) in index)
ambiguous = sum(1 for v in index.values() if len(v) > 1)
print(f"{label:22s} {100 * hits / len(queries):5.1f}% "
f"keys {len(index):7,} ambiguous {ambiguous:6,}")Run this before adopting anyone's recipe. The ordering of the steps is stable across datasets; the size of each step is not, and knowing which one carries your data tells you where to spend effort.
Explanation
Why exact matching fails at 29% rather than at 95%
The intuition that string matching "mostly works" comes from clean examples. Real address and place data varies along six axes at once — case, accents, punctuation, whitespace, abbreviation and typos — and a single difference on any axis is a complete miss.
The measured perturbations were each individually trivial. That is the point: no human would call any of them a different address, and string equality calls all of them different.
Why collapsing punctuation and whitespace carried the largest gain
Punctuation and whitespace variation is invisible on screen. A trailing space, a double space, a hyphen instead of a space, a comma instead of a hyphen — nobody sees these in a spreadsheet, so nobody fixes them by hand, so they accumulate.
Case and accents are half-visible: people notice LONDON and sometimes fix it. Abbreviations are fully visible, so they are usually already consistent by the time the file reaches you — which is precisely why expanding them added a tenth of a point.
Why fuzzy matching is a last resort rather than a strategy
Fuzzy matching is genuinely effective: 188 of 200 recovered at a 0.85 cutoff. It is also 40,000 times slower per lookup than a dictionary hit, and it introduces a threshold that has to be tuned, which means it introduces false matches.
The tiered design puts it where it earns its cost — on the 14.5% residue — and keeps the cheap path cheap. It also makes the threshold a decision about a small, inspectable set of rows rather than about the whole file.
Why the tiebreaker is a data problem, not an algorithm problem
When two records normalise to the same key, the strings carry no remaining information. Every additional comparison is arithmetic on identical inputs.
The only thing that can resolve it is a field that differs: postcode, region, population, an approximate coordinate. This is why address matching is really record linkage, and why the quality of the match depends far more on which fields you have than on which string algorithm you chose.
Edge cases or notes
- Blocking on a misspelled field silently loses true matches. Block on the most reliable field you have, and consider two passes with different blocks.
- Phonetic keys help for names and hurt for codes. Soundex on a postcode is noise.
- A 0.85 cutoff is a starting point, not a constant. Calibrate on labelled pairs; the right value depends on name length.
- Short names match everything. Guard against fuzzy-matching strings under about five characters.
- Transliteration is not normalisation.
MünchenandMunichare the same city and no accent fold connects them; that needs an alternate-names table. - Record the tier. An audit that cannot distinguish an exact match from a 0.86 fuzzy match is not an audit.
- One-to-many is the normal outcome, not an error. 10.9% of normalised keys had more than one record behind them.
- Never match on the raw string and the normalised string in the same join — you will double-count.
Internal links
- How to parse and normalise addresses in Python — the normaliser used here
- How to match addresses against a reference file — this design, implemented end to end
- How to deduplicate an address list in Python — the same machinery pointed at one file
- How to fuzzy match place names in Python — the fuzzy tier in more depth
- The address data model: why an address is not a string — the fields that break ties
- Match quality explained: reading a geocoder's confidence score — the same ambiguity, inside a geocoder
- Fixing a geocoder that matches the wrong town — what an unresolved tie looks like downstream
- How to merge near-duplicate features — the geometric version of this problem
FAQ
Why does joining on the address column miss so many rows?
Because the same address is written many ways. Measured: exact string equality matched 29.1% of realistically varied queries; normalisation took it to 85.5%.
Which normalisation step matters most?
Collapsing punctuation and whitespace, which added 27 points in the measured ladder, followed by case folding at 26. Abbreviation expansion added 0.1.
Does normalisation cause wrong matches?
A few. 10.9% of normalised keys covered more than one distinct place, and 0.8% of matches landed on a different place with the same name. That is the price of a 56-point recall gain.
When should I use fuzzy matching?
Only on the rows normalisation could not place — 14.5% in the measured run. It recovered 94% of those, at 134 ms per lookup against 3.3 microseconds for a dictionary hit.
How do I choose a blocking key?
Something cheap, discriminating and reliably present: the outward postcode, the first characters of the normalised name, or the country. Check the block-size distribution — a huge block means the key is missing for those rows.
What do I do with ambiguous matches?
Break the tie with a second field — postcode, region, population — and record which one you used. If nothing resolves it, report it rather than picking the first candidate.