The Address Data Model: Why an Address Is Not a String #

Problem statement #

An address arrives in a CSV as one column of text and gets treated as one value. Then somebody asks a question that the column cannot answer:

• how many customers are on the same street?
• which of these two rows are the same building?
• why did 12% of the file fail to geocode?

All three questions need the address broken into parts, and all three are unanswerable while it is a single string. Worse, the string is not even a stable representation of the place: Flat 2, 14 High St., 14a High Street, 14 HIGH STREET FLAT 2 and 14 High Street, Apartment 2 can all be the same front door, and none of them are equal to each other.

The measured cost of pretending otherwise: matching 5,000 lightly perturbed place names against a 238,483-row reference gazetteer, exact string equality matched 29.1%. The same comparison after decomposing and normalising matched 85.5%.

Quick answer #

Model an address as a record of typed components, not as text. The minimum useful set:

from dataclasses import dataclass, field


@dataclass
class Address:
    # sub-building, building, street: the three that decide identity
    unit: str | None = None            # Flat 2, Apt 4B, Suite 100
    house_number: str | None = None    # 14, 14a, 221B
    street: str | None = None          # High Street
    # locality: what disambiguates the street
    locality: str | None = None        # Camden Town
    town: str | None = None            # London
    region: str | None = None          # Greater London
    postcode: str | None = None        # NW1 8QP
    country: str = "GB"
    # provenance: never thrown away
    raw: str = ""
    parse_confidence: float | None = None

    def key(self) -> tuple:
        """The identity of the front door, for joining and deduplication."""
        return (self.country, (self.postcode or "").upper().replace(" ", ""),
                (self.street or "").lower(), (self.house_number or "").lower(),
                (self.unit or "").lower())

The key() method is the point of the whole exercise: two rows are the same address when their keys are equal, and no amount of string cleaning gives you that from one column.

Step-by-step solution #

1. Separate the three jobs the string is doing #

An address line carries three different kinds of information, and they behave differently:

A deduplication that includes the town in the key will split 14 High Street, Camden from 14 High Street, London even though they are the same door. A geocoder that ignores the town will return the wrong High Street out of the several hundred in the country.

2. Accept that the schema is country-specific #

There is no universal address schema, and any model that claims to be one is a European or American model with optional fields bolted on. The differences are structural, not cosmetic:

• House number position. 14 High Street in the UK, High Street 14 in Germany, 14 rue de la Paix in France.
• Number formats. 221B, 14a, 12-14, 4/7 (a unit-building compound in Australia), 1-2-3 (a Japanese block address that has no street at all).
• Administrative depth. Japan and Korea address by nested blocks; large parts of the world have no formal street addressing.
• Postcodes. Six characters in the UK, five or nine digits in the US, absent in Ireland before 2015 and in several countries still.

The practical consequence: store country first and branch parsing on it. A parser tuned for one country applied to another does not fail loudly, it fails by putting the street name in the locality field.

3. Normalise, but keep the original #

Normalisation is a lossy transformation applied for matching. Applied in place, it destroys evidence:

raw:        "Flat 2, 14 High St., CAMDEN, NW1 8QP"
normalised: unit="flat 2" number="14" street="high street" postcode="NW18QP"

Keep both. The normalised form is what you join on; the raw form is what you show a human when the join produced something surprising, and what you re-parse when the parser improves.

4. Treat the postcode as a strong but separate signal #

The postcode is the most useful single field in most address files: it is short, standardised, checkable against a published list, and independently sourced. It is also the field most often wrong, because it is the one people type from memory.

Use it as a check, not as the identity. If the parsed town and the postcode disagree, that is a flag on the record rather than a reason to overwrite either.

5. Record how confident the parse was #

A parser that returns components with no confidence forces every downstream step to assume the parse was right. Return a score — even a crude one, such as the fraction of the input tokens that were assigned to a component — and let the pipeline route low-confidence rows to review instead of into the map.

Code examples #

Example 1 — a rule-based parser for one country #

import re

STREET_TYPES = {
    "st": "street", "str": "street", "rd": "road", "ave": "avenue",
    "av": "avenue", "ln": "lane", "dr": "drive", "cl": "close",
    "ct": "court", "pl": "place", "sq": "square", "cres": "crescent",
}
UNIT_WORDS = r"(?:flat|apt|apartment|unit|suite|room|floor)"
UK_POSTCODE = re.compile(r"\b([A-Z]{1,2}\d[A-Z\d]?)\s*(\d[A-Z]{2})\b", re.I)


def parse_uk(raw: str) -> dict:
    """Pull components out of a UK address line. Order of removal matters:
    take the unambiguous things out first so what is left is smaller."""
    out = {"raw": raw}
    text = " ".join(raw.split())

    m = UK_POSTCODE.search(text)
    if m:
        out["postcode"] = f"{m.group(1).upper()} {m.group(2).upper()}"
        text = text[: m.start()] + text[m.end():]

    m = re.search(rf"\b{UNIT_WORDS}\.?\s*([\w\-/]+)", text, re.I)
    if m:
        out["unit"] = m.group(0).strip(" ,.")
        text = text[: m.start()] + text[m.end():]

    parts = [p.strip() for p in text.split(",") if p.strip()]
    if parts:
        m = re.match(r"^(\d+[a-z]?(?:-\d+[a-z]?)?)\s+(.*)$", parts[0], re.I)
        if m:
            out["house_number"], street = m.group(1), m.group(2)
        else:
            street = parts[0]
        tokens = street.lower().split()
        if tokens:
            tokens[-1] = STREET_TYPES.get(tokens[-1].strip("."), tokens[-1])
        out["street"] = " ".join(tokens)
        if len(parts) > 1:
            out["town"] = parts[-1]
        if len(parts) > 2:
            out["locality"] = parts[-2]

    assigned = sum(len(str(v).split()) for k, v in out.items() if k != "raw")
    out["parse_confidence"] = round(min(1.0, assigned / max(1, len(raw.split()))), 2)
    return out

>>> parse_uk("Flat 2, 14 High St., Camden, London, NW1 8QP")
{'raw': 'Flat 2, 14 High St., Camden, London, NW1 8QP',
 'postcode': 'NW1 8QP', 'unit': 'Flat 2', 'house_number': '14',
 'street': 'high street', 'town': 'London', 'locality': 'Camden',
 'parse_confidence': 1.0}

A regex parser is a starting point, not a destination. It is good enough to normalise a national file and bad at anything unusual — which is exactly why it must report confidence.

Example 2 — the identity key, and what it costs to get wrong #

import pandas as pd


def address_key(row) -> tuple:
    postcode = (row.get("postcode") or "").upper().replace(" ", "")
    return (
        row.get("country", "GB"),
        postcode,
        (row.get("street") or "").lower(),
        (row.get("house_number") or "").lower().lstrip("0"),
        (row.get("unit") or "").lower(),
    )


def duplicate_report(df: pd.DataFrame) -> pd.DataFrame:
    keys = df.apply(address_key, axis=1)
    dupes = keys[keys.duplicated(keep=False)]
    print(f"{len(df):,} rows -> {keys.nunique():,} distinct addresses")
    print(f"{len(dupes):,} rows share a key with another row")
    return df.assign(address_key=keys)

The lstrip("0") is not decoration. Files that have been through a spreadsheet routinely carry 007 where the source had 7, and a key that treats those as different addresses silently splits a customer in two.

Example 3 — validating components against each other #

def cross_check(a: dict, postcode_areas: dict) -> list[str]:
    """Return the reasons this parsed address should not be trusted."""
    problems = []
    if a.get("postcode"):
        area = a["postcode"].split()[0]
        expected_town = postcode_areas.get(area)
        if expected_town and a.get("town") and expected_town.lower() != a["town"].lower():
            problems.append(f"postcode {area} is {expected_town}, not {a['town']}")
    if a.get("house_number") and not a.get("street"):
        problems.append("house number with no street")
    if not a.get("postcode") and not a.get("town"):
        problems.append("no locality of any kind — cannot be disambiguated")
    if a.get("parse_confidence", 1) < 0.6:
        problems.append(f"low parse confidence {a['parse_confidence']}")
    return problems

Cross-checks between components are what turn a parser into a validator. Any single field can be wrong; two fields disagreeing is evidence.

Explanation #

Why exact matching fails so badly #

The 29.1% figure at the top comes from a controlled experiment: take a real place name, apply one realistic perturbation — uppercase it, strip its accents, abbreviate Saint to St, drop a character, replace a hyphen with a space, add stray whitespace — then look it up.

Every one of those perturbations is something a human types or a spreadsheet does. Individually they are trivial. Collectively they break seven lookups in ten, because string equality has no notion of "nearly".

Normalisation — lowercase, strip accents, collapse punctuation and whitespace, expand abbreviations — recovered the match rate to 85.5%. Sorting the tokens as well added nothing (85.5% again), because word order in addresses is not the problem; representation is.

Why the remaining 15% is the interesting part #

The residue after normalisation is typos and genuinely missing reference data. Fuzzy matching recovered 188 of 200 attempts at a 0.85 similarity cutoff — but at 134 ms per lookup against 3.3 microseconds for a dictionary hit.

That ratio is the design rule for every address pipeline: normalise everything, fuzzy-match only the residue. Fuzzy matching a whole file is not a better algorithm, it is the same algorithm run a thousand times more expensively on rows that did not need it.

Why normalisation creates its own ambiguity #

Normalisation merges. Of 190,028 normalised place-name keys in the reference set, 20,765 — 10.9% — map to more than one distinct place. Removing accents alone creates 47,607 collisions inside a single country, because Saint-Étienne and Saint-Etienne are the same place but Ávila and Avila are not always.

So a normalised match is not automatically a correct match. In the same experiment, 38 of the matched rows pointed at a different place with the same normalised name. That is a 0.8% wrong-target rate — small, invisible, and permanent once it is in the database.

Why the unit belongs in the key and the town does not #

Identity is the smallest set of fields that distinguishes one deliverable location from another. Flat 2 and Flat 3 are different locations; Camden and London are two names for the context of the same location.

Putting context in the key splits records that should join. Leaving the unit out of the key merges records that should not. Both errors are silent, and the second one is worse, because it produces a smaller, cleaner-looking table.

Edge cases or notes #

• Many places have no street address. Japan addresses by block; large parts of the world have no formal addressing at all. A schema that requires street will reject them.
• PO boxes are not locations. Geocode them and you get the sorting office.
• St is ambiguous — Street or Saint. Expand it using position: leading St is usually Saint, trailing St is usually Street.
• Ranges and compounds — 12-14 High Street, 4/7 Smith St — need a rule, not a parser. Decide whether they are one address or two and record which.
• Case is not free. Turkish dotless ı breaks naive .lower(); use casefold and be careful with locale.
• 22.5% of place names contain non-ASCII characters. Any pipeline that assumes ASCII will lose them or mangle them.
• Keep country first in the key. It is the only field that changes the meaning of all the others.
• Never overwrite the raw string. It is the only thing you can re-parse when the parser improves.

Internal links #

• Geocoding explained: from an address string to a coordinate — what happens after parsing
• How to parse and normalise addresses in Python — the working implementation
• Address matching explained: why exact string equality fails — the matching side in detail
• How to deduplicate an address list in Python — using the identity key
• How to clean CSV coordinates into a GeoDataFrame — the coordinate side of the same file
• How to standardise attribute values — the general normalisation pattern
• Spatial data quality dimensions — where address quality sits in the wider picture
• Fixing addresses that break on accents and encoding — when normalisation is the problem

FAQ #

Why can I not just use the full address string as a key? #

Because the same door is written several ways. In a controlled test, exact string equality matched 29.1% of lightly perturbed queries; the same data decomposed and normalised matched 85.5%.

What are the minimum fields an address model needs? #

Country, postcode, street, house number and unit for identity; town and region for context; and the raw string plus a parse confidence for provenance.

Should the postcode be part of the identity key? #

Yes, as a strong disambiguator for the street — but check it against the town rather than trusting it. It is the field people most often mistype.

Is there a standard address schema I can adopt? #

There are several — OASIS xAL, the OpenAddresses schema, ISO 19160 — and all of them are supersets that make almost every field optional. They are useful as a checklist, not as a table definition.

Do I need a machine-learning parser? #

Only if your data is genuinely international and unstructured. A rule-based parser per country handles national files well and is debuggable; statistical parsers such as libpostal earn their place on mixed multi-country input.

How do I handle addresses with no street? #

Keep the model but leave street empty and rely on locality plus a local identifier. Do not force the block or village name into the street field — that is what breaks the geocoder later.