🔧 Fix Geocoding & addresses Files & formats Geocoding & Addresses · step 19 of 20 Updated

Fixing Addresses That Break on Accents and Encoding

Problem statement

The address file looks like this:

Münchener Straße 14
Ã…lesund
Café Rouge, Rue de l'Église

Those are München, Ålesund and Café Rouge, Rue de l'Église, encoded as UTF-8 and then decoded as Latin-1 — the classic mojibake. Every one of them will fail to geocode, fail to match a reference file, and sort incorrectly.

The scale is not marginal. In a gazetteer of 5,226,942 populated places, 1,175,104 names — 22.5% — contain non-ASCII characters, and 1,183,124 records have a name that differs from their asciiname. Any pipeline that assumes ASCII loses or mangles roughly a fifth of the world.

There are three distinct problems hiding under "accents are broken", and they need different fixes: the bytes were decoded wrongly, the characters are composed differently, or the accents were stripped and created collisions.

Quick answer

Diagnose first — a repair applied to the wrong problem makes it worse:

import unicodedata


def diagnose_text(s: str) -> str:
    if not isinstance(s, str):
        return "not a string"
    if any(marker in s for marker in ("Ã", "Â", "â€", "")):
        return "mojibake: UTF-8 bytes decoded as Latin-1"
    if "�" in s:
        return "replacement characters: information already lost"
    if s != unicodedata.normalize("NFC", s):
        return "decomposed form (NFD): looks identical, compares unequal"
    if all(ord(c) < 128 for c in s):
        return "pure ASCII"
    return "clean non-ASCII"


def repair_mojibake(s: str) -> str:
    """Reverse the classic double-decode. Windows text needs cp1252, not latin-1:
    the byte 0x9f is undefined in Latin-1 but is the character 'Ÿ' in CP1252,
    which is exactly what a mangled 'ß' turns into."""
    for codec in ("cp1252", "latin-1"):
        try:
            fixed = s.encode(codec).decode("utf-8")
        except (UnicodeEncodeError, UnicodeDecodeError):
            continue                      # not mojibake of this kind — leave it alone
        return fixed
    return s
>>> repair_mojibake("Münchener Straße 14")
'Münchener Straße 14'
>>> repair_mojibake("Café Rouge, Rue de l’Église")
'Café Rouge, Rue de l’Église'
>>> repair_mojibake("Łódź")          # not mojibake — returned untouched
'Łódź'
Triage table of four text-encoding symptoms and the fix for each.
Only the first is reversible by code; the second needs new data.

Step-by-step solution

1. Find where the encoding was lost

Mojibake happens at a boundary. Walk back through the pipeline and find the first place the text is wrong:

  • The CSV read. pd.read_csv without encoding= uses the platform default, which on some systems is not UTF-8.
  • The shapefile's .dbf. Shapefiles carry an optional .cpg file naming the encoding; without it, readers guess.
  • The database connection. A client encoding mismatch corrupts on read or write.
  • The export from the source system. Frequently the real culprit, and the only place a permanent fix is possible.

Read ten raw bytes and look:

with open("addresses.csv", "rb") as f:
    raw = f.read(2000)
print(raw[:200])
print("BOM:", raw[:3] == b"\xef\xbb\xbf")

b'M\xc3\xbcnchen' is correct UTF-8 for München. b'M\xfcnchen' is Latin-1. b'M\xc3\x83\xc2\xbcnchen' is UTF-8 that was already mojibake before it was written.

2. Repair only if the round trip is clean

s.encode("cp1252").decode("utf-8") reverses the classic double-decoding. It raises on text that was not damaged that way — Łódź cannot be encoded as CP1252 at all — which is the safety property that makes it usable in bulk:

repaired = df["address"].map(repair_mojibake)
changed = (repaired != df["address"]).sum()
print(f"repaired {changed:,} of {len(df):,} values")

Inspect a sample of the changes before committing. A file with mixed encodings — half the rows written by one system and half by another — needs the repair applied per row, exactly as above, and never to the whole file at once.

If the text contains � (the replacement character), the original bytes are gone. No repair is possible; the fix is a fresh export.

3. Normalise the Unicode form

é can be one code point (U+00E9) or two (e + U+0301). They render identically and compare unequal:

a = "Caf\u00e9"           # composed:   C a f é          (4 code points)
b = "Cafe\u0301"          # decomposed: C a f e + acute  (5 code points)

print(a, b, a == b)        # Café Café False
print(unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b))   # True

Decomposed forms arrive routinely from macOS filesystems and some web forms. Normalise to NFC on input — one line, applied everywhere text enters the system — and the whole class of "identical strings that do not match" disappears.

4. Fold accents for matching, not for storage

Matching wants Saint-Étienne and Saint-Etienne to be equal. Storage wants the original.

Keep both columns. The folded form is the join key; the original is what you display, export and send to a geocoder. Folding in place is a permanent loss, and it is the loss that turns 22.5% of your place names into approximations.

5. Know what folding costs

Folding merges. In the same gazetteer, removing accents created 47,607 collisions within a single country — pairs of genuinely different places whose ASCII forms are identical.

So a folded match needs a tiebreaker, exactly like any other normalised match: postcode, region, population. Folding is a recall device with a precision cost, not a cleanup.

6. Send the right form to the geocoder

Geocoders index both forms and generally handle either. What they do not handle is mojibake: München is not a place, and the search will return nothing at all.

Repair before geocoding, and send the composed original rather than the folded form — the geocoder's own normalisation is better informed than yours, because it knows which language the name is in.

Bar chart showing 22.5% of place names contain non-ASCII characters.
A repair pass that drives non-ASCII to zero has destroyed data, not cleaned it.

Code examples

Example 1 — an encoding audit of a whole file 

import pandas as pd
import unicodedata
from collections import Counter


def encoding_audit(df: pd.DataFrame, columns=None) -> pd.DataFrame:
    columns = columns or [c for c in df.columns if df[c].dtype == object]
    rows = []
    for col in columns:
        values = df[col].dropna().astype(str)
        if values.empty:
            continue
        rows.append({
            "column": col,
            "non_ascii": int(values.map(lambda s: any(ord(c) > 127 for c in s)).sum()),
            "mojibake": int(values.str.contains(r"Ã.|Â.|â€", regex=True).sum()),
            "replacement_char": int(values.str.contains("�").sum()),
            "decomposed": int(values.map(
                lambda s: s != unicodedata.normalize("NFC", s)).sum()),
            "rows": len(values),
        })
    audit = pd.DataFrame(rows).set_index("column")
    audit["pct_non_ascii"] = (100 * audit["non_ascii"] / audit["rows"]).round(1)
    return audit
            non_ascii  mojibake  replacement_char  decomposed   rows  pct_non_ascii
address         8,912     8,912                 0           0  40,000           22.3
town            2,104     2,104                 0           0  40,000            5.3
name           11,338         0                 0       1,204  40,000           28.3

Two different problems in one file: the address and town columns are mojibake; the name column is clean but partly decomposed.

Example 2 — a safe repair pass with a report 

def repair_column(series: pd.Series, sample=8) -> tuple[pd.Series, pd.DataFrame]:
    repaired = series.map(lambda s: repair_mojibake(s) if isinstance(s, str) else s)
    changed = repaired != series
    normalised = repaired.map(
        lambda s: unicodedata.normalize("NFC", s) if isinstance(s, str) else s)
    renormalised = (normalised != repaired) & ~changed

    report = pd.DataFrame({"before": series[changed], "after": repaired[changed]}).head(sample)
    print(f"mojibake repaired : {int(changed.sum()):,}")
    print(f"NFC-normalised    : {int(renormalised.sum()):,}")
    print(f"still non-ASCII   : "
          f"{int(normalised.map(lambda s: isinstance(s, str) and any(ord(c) > 127 for c in s)).sum()):,}"
          f"   (expected — about a fifth of place names are)")
    return normalised, report

The third line matters. Non-ASCII text is not a problem to be eliminated; a repair pass that drives it to zero has destroyed data.

Example 3 — reading files with the right encoding in the first place 

import pandas as pd


def read_csv_safely(path, **kwargs):
    """Try the encodings that actually occur, in the order they occur."""
    for encoding in ("utf-8-sig", "utf-8", "cp1252", "latin-1"):
        try:
            df = pd.read_csv(path, encoding=encoding, **kwargs)
            print(f"read as {encoding}")
            if encoding in ("cp1252", "latin-1"):
                print("  ! this file is not UTF-8; ask the source to export UTF-8")
            return df
        except UnicodeDecodeError:
            continue
    raise ValueError(f"could not decode {path} with any candidate encoding")


def read_shapefile_safely(path):
    """Shapefiles carry their encoding in a .cpg file — when they carry it at all."""
    import geopandas as gpd
    from pathlib import Path

    cpg = Path(path).with_suffix(".cpg")
    if cpg.exists():
        print(f"declared encoding: {cpg.read_text().strip()}")
        return gpd.read_file(path)
    print("no .cpg file — attribute encoding is a guess; check the text columns")
    return gpd.read_file(path, encoding="utf-8")

latin-1 decodes any byte sequence without error, so it never raises — which is why it is last. It will happily produce mojibake from a UTF-8 file, silently.

Explanation

Why mojibake is reversible and replacement characters are not

Mojibake preserves the bytes and misinterprets them. München encoded as UTF-8 is the byte sequence 4d c3 bc 6e 63 68 65 6e; decoded as CP1252 those bytes become the characters München. Re-encoding as CP1252 recovers the original bytes exactly, and decoding those as UTF-8 recovers the original text.

The codec matters. Windows text is CP1252, not Latin-1, and the two differ in the 0x80–0x9F range — where a mangled ß lands. "Straße".encode("latin-1") raises, because Ÿ has no Latin-1 byte; .encode("cp1252") succeeds and the round trip completes.

A replacement character (�) is different: the decoder met a byte sequence it could not interpret and substituted a marker. The original bytes were discarded. No function recovers them, and the only fix is a new export from the source.

Why NFC normalisation belongs at the boundary

Unicode allows the same character to be written as one code point or as a base plus combining marks. Both are valid, both render identically, and they are unequal to every string comparison, every dictionary key and every database index.

Normalising on input rather than at each comparison means the invariant holds everywhere afterwards: any two strings that look the same are the same. Doing it at comparison time means remembering to do it, which fails eventually.

Why accent folding is a matching device and not a cleanup

Folding is lossy in a way that matters. Ávila and Avila, São Paulo and Sao Paulo are the same places written with and without accents — but Malmö and Malmo being merged is fine while some other pair being merged is not, and the function cannot tell the difference.

Measured: folding created 47,607 collisions inside a single country. That is the cost of the recall gain, and it is acceptable only because the folded form is a key, sitting alongside an intact original that can break the tie.

Why the source export is the real fix

Every repair described here is a workaround for text that was correct once. Each run of the pipeline re-applies it, and each new file risks a slightly different corruption that the heuristic does not catch.

If the file comes from a system you or a colleague controls, the durable fix is one setting on the export: UTF-8, with a BOM if the consumer is a spreadsheet. Ten minutes there removes a permanent line of defensive code.

Two panels showing the recall gain and the collision cost of accent folding.
Folding merges. That is the point, and it is also the risk.

Edge cases or notes

  • Try cp1252 before latin-1 when reversing mojibake; the two differ exactly where Windows punctuation and Ÿ live.
  • utf-8-sig strips the byte-order mark; plain utf-8 leaves it on the first column name.
  • Excel writes UTF-8 with a BOM and reads UTF-8 without one badly. utf-8-sig on both sides is the pragmatic choice.
  • Shapefile .dbf encoding is declared in .cpg, which is optional and often absent or wrong.
  • Turkish I does not lowercase to i. casefold() handles most cases; locale-correct casing needs a locale-aware library.
  • German ß casefolds to ss, which is what you want for matching and not for display.
  • Do not fold in place. Keep the original; the folded form is a derived column.
  • Non-ASCII is normal. 22.5% of place names contain it; a pipeline that treats it as an error is the error.
  • Test with a fixture containing München, Ålesund, Łódź, İzmir and Kraków — those five break most naive code.

FAQ

What causes München instead of München?

UTF-8 bytes decoded as Latin-1 or CP1252. It is reversible: encode back to Latin-1 and decode as UTF-8. If the string contains replacement characters instead, the bytes are gone and only a new export will fix it.

Should I strip accents from my addresses?

Only into a separate matching key. Folding merged 47,607 genuinely distinct place names within one country in a measured test, so the original has to survive to break the ties.

What is NFC normalisation and do I need it?

It is the composed Unicode form. é can be one code point or two that render identically and compare unequal. Normalise to NFC at the point text enters your system and the problem disappears.

How much of my data will contain non-ASCII characters?

In a global gazetteer, 22.5% of populated-place names do. Any pipeline that treats non-ASCII as an error will damage roughly a fifth of the world's place names.

Why does reading a shapefile give me the wrong characters?

The .dbf encoding is declared in an optional .cpg file. Without it, readers guess — usually Latin-1 — and UTF-8 attributes come back as mojibake.

Can I just read everything as Latin-1?

No. Latin-1 decodes any byte sequence without raising, so it silently turns valid UTF-8 into mojibake. That is why it belongs last in a fallback chain, if at all.