Fixing Addresses That Break on Accents and Encoding

On this page

Problem statement

The address file looks like this:

MÃ¼nchener StraÃŸe 14
Ã…lesund
CafÃ© Rouge, Rue de l'Ã‰glise

Those are München, Ålesund and Café Rouge, Rue de l'Église, encoded as UTF-8 and then decoded as Latin-1 — the classic mojibake. Every one of them will fail to geocode, fail to match a reference file, and sort incorrectly.

The scale is not marginal. In the GeoNames gazetteer's 5,231,962 populated places, 1,175,491 names — 22.5% — contain non-ASCII characters, and 1,183,508 records have a name that differs from their asciiname. Any pipeline that assumes ASCII loses or mangles roughly a fifth of the world.

There are three distinct problems hiding under "accents are broken", and they need different fixes: the bytes were decoded wrongly, the characters are composed differently, or the accents were stripped and created collisions.

Quick answer

Diagnose first — a repair applied to the wrong problem makes it worse:

import unicodedata


def diagnose_text(s: str) -> str:
    if not isinstance(s, str):
        return "not a string"
    if any(marker in s for marker in ("Ã", "Â", "â€", "Ã¯Â»Â¿")):
        return "mojibake: UTF-8 bytes decoded as Latin-1"
    if "�" in s:
        return "replacement characters: information already lost"
    if s != unicodedata.normalize("NFC", s):
        return "decomposed form (NFD): looks identical, compares unequal"
    if all(ord(c) < 128 for c in s):
        return "pure ASCII"
    return "clean non-ASCII"


def repair_mojibake(s: str) -> str:
    """Reverse the classic double-decode. Windows text needs cp1252, not latin-1:
    the byte 0x9f is undefined in Latin-1 but is the character 'Ÿ' in CP1252,
    which is exactly what a mangled 'ß' turns into."""
    for codec in ("cp1252", "latin-1"):
        try:
            fixed = s.encode(codec).decode("utf-8")
        except (UnicodeEncodeError, UnicodeDecodeError):
            continue                      # not mojibake of this kind — leave it alone
        return fixed
    return s

for text in ("MÃ¼nchener StraÃŸe 14", "CafÃ© Rouge, Rue de lâ€™Ã‰glise", "Łódź"):
    print(f"{text!r:40} -> {repair_mojibake(text)!r:32} {diagnose_text(text)}")

'MÃ¼nchener StraÃŸe 14'                  -> 'Münchener Straße 14'            mojibake: UTF-8 bytes decoded as Latin-1
'CafÃ© Rouge, Rue de lâ€™Ã‰glise'        -> 'Café Rouge, Rue de l’Église'    mojibake: UTF-8 bytes decoded as Latin-1
'Łódź'                                   -> 'Łódź'                           clean non-ASCII

Only the first is reversible by code; the second needs new data.

Find where encoding was lost, repair clean round trips, normalise

1. Find where the encoding was lost

Mojibake happens at a boundary. Walk back through the pipeline and find the first place the text is wrong:

The CSV read. pd.read_csv without encoding= uses the platform default, which on some systems is not UTF-8.
The shapefile's .dbf. Shapefiles carry an optional .cpg file naming the encoding; without it, readers guess.
The database connection. A client encoding mismatch corrupts on read or write.
The export from the source system. Frequently the real culprit, and the only place a permanent fix is possible.

Read ten raw bytes and look:

with open("addresses.csv", "rb") as f:
    raw = f.read(2000)
print(raw[:200])
print("BOM:", raw[:3] == b"\xef\xbb\xbf")

b'M\xc3\xbcnchen' is correct UTF-8 for München. b'M\xfcnchen' is Latin-1. b'M\xc3\x83\xc2\xbcnchen' is UTF-8 that was already mojibake before it was written.

2. Repair only if the round trip is clean

s.encode("cp1252").decode("utf-8") reverses the classic double-decoding. It raises on text that was not damaged that way — Łódź cannot be encoded as CP1252 at all — which is the safety property that makes it usable in bulk:

repaired = df["address"].map(repair_mojibake)
changed = (repaired != df["address"]).sum()
print(f"repaired {changed:,} of {len(df):,} values")

Inspect a sample of the changes before committing. A file with mixed encodings — half the rows written by one system and half by another — needs the repair applied per row, exactly as above, and never to the whole file at once.

If the text contains � (the replacement character), the original bytes are gone. No repair is possible; the fix is a fresh export.

3. Normalise the Unicode form

é can be one code point (U+00E9) or two (e + U+0301). They render identically and compare unequal:

a = "Caf\u00e9"           # composed:   C a f é          (4 code points)
b = "Cafe\u0301"          # decomposed: C a f e + acute  (5 code points)

print(a, b, a == b)        # Café Café False
print(unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b))   # True

Decomposed forms arrive routinely from macOS filesystems and some web forms. Normalise to NFC on input — one line, applied everywhere text enters the system — and the whole class of "identical strings that do not match" disappears.

4. Fold accents for matching, not for storage

Matching wants Saint-Étienne and Saint-Etienne to be equal. Storage wants the original.

Keep both columns. The folded form is the join key; the original is what you display, export and send to a geocoder. Folding in place is a permanent loss, and it is the loss that turns 22.5% of your place names into approximations.

5. Know what folding costs

Folding merges. In the same gazetteer, removing accents and case merged 55,598 distinct place names into another name in the same country, in 48,754 groups — genuinely different spellings whose folded forms are identical.

So a folded match needs a tiebreaker, exactly like any other normalised match: postcode, region, population. Folding is a recall device with a precision cost, not a cleanup.

6. Send the right form to the geocoder

Geocoders index both forms and generally handle either. What they do not handle is mojibake: MÃ¼nchen is not a place, and the search will return nothing at all.

Repair before geocoding, and send the composed original rather than the folded form — the geocoder's own normalisation is better informed than yours, because it knows which language the name is in.

A repair pass that drives non-ASCII to zero has destroyed data, not cleaned it.

An encoding audit and a safe repair pass

The data used on this page

The GeoNames allCountries dump, filtered to populated places (feature class P). The damaged file in Examples 1 and 2 is built from it on purpose: 40,000 populated-place names drawn with a fixed seed, with one column passed through the classic UTF-8-read-as-CP1252 mis-decode and one in ten names written in decomposed (NFD) form, as a macOS filesystem would. Because the damage is known, the audit and the repair can be checked against the right answer.

import hashlib
import os
import unicodedata
import urllib.request
import zipfile

import pandas as pd

if not os.path.exists("allCountries.zip"):
    urllib.request.urlretrieve("https://download.geonames.org/export/dump/allCountries.zip", "allCountries.zip")
cols = ["geonameid", "name", "asciiname", "feature_class", "country"]
with zipfile.ZipFile("allCountries.zip") as z, z.open("allCountries.txt") as f:
    gaz = pd.read_csv(f, sep="\t", header=None, usecols=[0, 1, 2, 6, 8], names=cols, dtype=str,
                      quoting=3, keep_default_na=False)
places = gaz[gaz.feature_class == "P"]
non_ascii = places.name.map(lambda s: not s.isascii())
print(f"{len(places):,} populated places; {non_ascii.sum():,} names ({non_ascii.mean():.1%}) contain non-ASCII; "
      f"{(places.name != places.asciiname).sum():,} differ from asciiname")


def fold(s):
    return "".join(c for c in unicodedata.normalize("NFKD", s) if not unicodedata.combining(c)).casefold()


folded = places.assign(key=places.name.map(fold))
distinct = folded.drop_duplicates(["country", "name"])
groups = distinct.groupby(["country", "key"]).name.nunique()
print(f"folding merges {int((groups[groups > 1] - 1).sum()):,} distinct names into another name "
      f"in the same country, in {int((groups > 1).sum()):,} groups")

# a fixed 40,000: the places whose ids hash lowest, so GeoNames' daily edits barely move the sample
order = places.geonameid.map(lambda i: hashlib.sha256(i.encode()).hexdigest())
sample = places.loc[order.sort_values().index[:40_000]].reset_index(drop=True)
nfd = sample.index % 10 == 0
test = pd.DataFrame({
    "original": sample.name,
    "mangled": sample.name.map(lambda s: s.encode("utf-8").decode("cp1252", errors="replace")),
    "name": [unicodedata.normalize("NFD", s) if d else s for s, d in zip(sample.name, nfd)],
})

5,231,962 populated places; 1,175,491 names (22.5%) contain non-ASCII; 1,183,508 differ from asciiname
folding merges 55,598 distinct names into another name in the same country, in 48,754 groups

Example 1 — an encoding audit of a whole file

import pandas as pd
import unicodedata
from collections import Counter


def encoding_audit(df: pd.DataFrame, columns=None) -> pd.DataFrame:
    # pandas 3 stores text as the "str" dtype, not object: select both, or the audit sees no columns
    columns = columns or list(df.select_dtypes(include=["object", "string"]).columns)
    rows = []
    for col in columns:
        values = df[col].dropna().astype(str)
        if values.empty:
            continue
        rows.append({
            "column": col,
            "non_ascii": int(values.map(lambda s: any(ord(c) > 127 for c in s)).sum()),
            "mojibake": int(values.str.contains(r"Ã.|Â.|â€", regex=True).sum()),
            "replacement_char": int(values.str.contains("�").sum()),
            "decomposed": int(values.map(
                lambda s: s != unicodedata.normalize("NFC", s)).sum()),
            "rows": len(values),
        })
    audit = pd.DataFrame(rows).set_index("column")
    audit["pct_non_ascii"] = (100 * audit["non_ascii"] / audit["rows"]).round(1)
    return audit


print(encoding_audit(test).to_string())

          non_ascii  mojibake  replacement_char  decomposed   rows  pct_non_ascii
column                                                                           
original       8832         2                 0           0  40000           22.1
mangled        8832      4109              3586           0  40000           22.1
name           8832         1                 0         768  40000           22.1

Two different problems in one file. Every non-ASCII value in the mangled column is damaged, yet the mojibake pattern caught only 4,109 of the 8,832: mis-decoded Cyrillic, Greek or Arabic starts with Ð, Î or Ø, not Ã. Another 3,586 carry replacement characters, because five byte values have no CP1252 character at all. The pattern also flagged 2 genuine names. The name column is clean but 768 values are decomposed. Only rows that were non-ASCII to begin with show damage — the rest are byte-for-byte the same in UTF-8 and CP1252, which is why mojibake hides in files that are mostly English.

The repair is exact where the bytes survived and impossible where they did not.

Example 2 — a safe repair pass with a report

def repair_column(series: pd.Series, sample=8) -> tuple[pd.Series, pd.DataFrame]:
    repaired = series.map(lambda s: repair_mojibake(s) if isinstance(s, str) else s)
    changed = repaired != series
    normalised = repaired.map(
        lambda s: unicodedata.normalize("NFC", s) if isinstance(s, str) else s)
    renormalised = (normalised != repaired) & ~changed

    report = pd.DataFrame({"before": series[changed], "after": repaired[changed]}).head(sample)
    print(f"mojibake repaired : {int(changed.sum()):,}")
    print(f"NFC-normalised    : {int(renormalised.sum()):,}")
    print(f"still non-ASCII   : "
          f"{int(normalised.map(lambda s: isinstance(s, str) and any(ord(c) > 127 for c in s)).sum()):,}"
          f"   (expected — about a fifth of place names are)")
    return normalised, report


fixed, report = repair_column(test["mangled"])
print(f"repaired rows equal to the original names: {(fixed == test['original']).mean():.2%}")

mojibake repaired : 5,246
NFC-normalised    : 0
still non-ASCII   : 8,832   (expected — about a fifth of place names are)
repaired rows equal to the original names: 91.03%

The third line matters. Non-ASCII text is not a problem to be eliminated; a repair pass that drives it to zero has destroyed data. And the last line is the honest ceiling: 9.0% of the file — the 3,586 names that had picked up replacement characters, four in ten of the damaged ones — could not be restored, because those bytes were gone before the repair ran.

Example 3 — reading files with the right encoding in the first place

import pandas as pd


def read_csv_safely(path, **kwargs):
    """Try the encodings that actually occur, in the order they occur."""
    for encoding in ("utf-8-sig", "utf-8", "cp1252", "latin-1"):
        try:
            df = pd.read_csv(path, encoding=encoding, **kwargs)
            print(f"read as {encoding}")
            if encoding in ("cp1252", "latin-1"):
                print("  ! this file is not UTF-8; ask the source to export UTF-8")
            return df
        except UnicodeDecodeError:
            continue
    raise ValueError(f"could not decode {path} with any candidate encoding")


def read_shapefile_safely(path):
    """Shapefiles carry their encoding in a .cpg file — when they carry it at all."""
    import geopandas as gpd
    from pathlib import Path

    cpg = Path(path).with_suffix(".cpg")
    if cpg.exists():
        print(f"declared encoding: {cpg.read_text().strip()}")
        return gpd.read_file(path)
    print("no .cpg file — attribute encoding is a guess; check the text columns")
    return gpd.read_file(path, encoding="utf-8")

latin-1 decodes any byte sequence without error, so it never raises — which is why it is last. It will happily produce mojibake from a UTF-8 file, silently.

Mojibake is reversible; replacement characters are not

Why mojibake is reversible and replacement characters are not

Mojibake preserves the bytes and misinterprets them. München encoded as UTF-8 is the byte sequence 4d c3 bc 6e 63 68 65 6e; decoded as CP1252 those bytes become the characters MÃ¼nchen. Re-encoding as CP1252 recovers the original bytes exactly, and decoding those as UTF-8 recovers the original text.

The codec matters. Windows text is CP1252, not Latin-1, and the two differ in the 0x80–0x9F range — where a mangled ß lands. "StraÃŸe".encode("latin-1") raises, because Ÿ has no Latin-1 byte; .encode("cp1252") succeeds and the round trip completes.

A replacement character (�) is different: the decoder met a byte sequence it could not interpret and substituted a marker. The original bytes were discarded. No function recovers them, and the only fix is a new export from the source.

Why NFC normalisation belongs at the boundary

Unicode allows the same character to be written as one code point or as a base plus combining marks. Both are valid, both render identically, and they are unequal to every string comparison, every dictionary key and every database index.

Normalising on input rather than at each comparison means the invariant holds everywhere afterwards: any two strings that look the same are the same. Doing it at comparison time means remembering to do it, which fails eventually.

Why accent folding is a matching device and not a cleanup

Folding is lossy in a way that matters. Ávila and Avila, São Paulo and Sao Paulo are the same places written with and without accents — but Malmö and Malmo being merged is fine while some other pair being merged is not, and the function cannot tell the difference.

Measured: folding merged 55,598 distinct place names into another within the same country. That is the cost of the recall gain, and it is acceptable only because the folded form is a key, sitting alongside an intact original that can break the tie.

Why the source export is the real fix

Every repair described here is a workaround for text that was correct once. Each run of the pipeline re-applies it, and each new file risks a slightly different corruption that the heuristic does not catch.

If the file comes from a system you or a colleague controls, the durable fix is one setting on the export: UTF-8, with a BOM if the consumer is a spreadsheet. Ten minutes there removes a permanent line of defensive code.

Folding merges. That is the point, and it is also the risk.

cp1252, byte-order marks and Excel

Try cp1252 before latin-1 when reversing mojibake; the two differ exactly where Windows punctuation and Ÿ live.
utf-8-sig strips the byte-order mark; plain utf-8 leaves it on the first column name.
Excel writes UTF-8 with a BOM and reads UTF-8 without one badly. utf-8-sig on both sides is the pragmatic choice.
Shapefile .dbf encoding is declared in .cpg, which is optional and often absent or wrong.
Turkish I does not lowercase to i. casefold() handles most cases; locale-correct casing needs a locale-aware library.
German ß casefolds to ss, which is what you want for matching and not for display.
Do not fold in place. Keep the original; the folded form is a derived column.
Non-ASCII is normal. 22.5% of place names contain it; a pipeline that treats it as an error is the error.
Test with a fixture containing München, Ålesund, Łódź, İzmir and Kraków — those five break most naive code.

Internal links

Fixing shapefile encoding errors in Python — the same problem inside a .dbf
How to parse and normalise addresses in Python — where folding belongs in the pipeline
Address matching explained: why exact string equality fails — what folding buys and costs
My geocoder returns no match for addresses that exist — mojibake as a cause of no-matches
How to standardise attribute values — the general text cleaning pattern
The address data model: why an address is not a string — keeping the original alongside the key
Fixing truncated shapefile column names — the other shapefile text limitation
How to clean CSV coordinates into a GeoDataFrame — the numeric half of the same file

FAQ

What causes `MÃ¼nchen` instead of `München`?

UTF-8 bytes decoded as Latin-1 or CP1252. It is reversible: encode back to Latin-1 and decode as UTF-8. If the string contains replacement characters instead, the bytes are gone and only a new export will fix it.

Should I strip accents from my addresses?

Only into a separate matching key. Folding merged 55,598 distinct GeoNames place names into another within the same country, so the original has to survive to break the ties.

What is NFC normalisation and do I need it?

It is the composed Unicode form. é can be one code point or two that render identically and compare unequal. Normalise to NFC at the point text enters your system and the problem disappears.

How much of my data will contain non-ASCII characters?

In a global gazetteer, 22.5% of populated-place names do. Any pipeline that treats non-ASCII as an error will damage roughly a fifth of the world's place names.

Why does reading a shapefile give me the wrong characters?

The .dbf encoding is declared in an optional .cpg file. Without it, readers guess — usually Latin-1 — and UTF-8 attributes come back as mojibake.

Can I just read everything as Latin-1?

No. Latin-1 decodes any byte sequence without raising, so it silently turns valid UTF-8 into mojibake. That is why it belongs last in a fallback chain, if at all.