Garbled Attribute Text and UnicodeDecodeError from a Shapefile: How to Fix Encoding

On this page

Problem statement

The place names came back wrong:

gdf = gpd.read_file("data/raw/orte_utf8.shp")
print(gdf["name"].head(3).to_list())

['FÃ¼rth', 'MÃ¼nster', 'DÃ¼sseldorf']

Or the read fails outright:

try:
    gpd.read_file("data/raw/orte_cp1252.shp", encoding="utf-8")
except UnicodeDecodeError as exc:
    print("UnicodeDecodeError:", exc)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 1: invalid start byte

Both files hold the same five German city names from Natural Earth. One was written as UTF-8 and the other as Windows-1252, and both have lost their .cpg sidecar — the everyday case of a shapefile mailed as "just the .shp, .shx and .dbf". A shapefile stores attributes in a dBase (.dbf) table, which has no reliable way to say what character encoding it used. There is the optional .cpg sidecar and a one-byte language driver code in the header, and both are frequently missing or wrong. The reader then has to guess, and a wrong guess produces either mojibake — Ã¼ where ü belongs — or a hard decode error.

How a shapefile ends up read in the wrong encoding:

no .cpg file, so GDAL falls back to its default of ISO-8859-1
the data was written as Windows-1252 or ISO-8859-1 by a desktop GIS
Central and Eastern European data written as Windows-1250, Cyrillic as Windows-1251
Chinese, Japanese or Korean data in GBK, Shift-JIS or EUC-KR
a file that has already been double-encoded by an earlier faulty conversion
the .cpg says UTF-8 but the bytes are not

Quick answer

To read a shapefile with the correct encoding:

look for a .cpg file next to the .shp and read it
try UTF-8 first; if it fails or looks wrong, try cp1252, then latin1
pass the encoding explicitly: gpd.read_file(path, encoding="cp1252")
verify by looking at the characters, not by the absence of an exception
write outputs as UTF-8 — GeoPackage always is — and stop the problem spreading

for path in (Path("data/raw/orte_utf8.shp"), Path("data/raw/orte_cp1252.shp")):
    cpg = path.with_suffix(".cpg")
    declared = cpg.read_text(encoding="ascii", errors="ignore").strip() if cpg.exists() else None
    print(f"{path.name}: declared encoding {declared or '<none>'}")
    for enc in filter(None, [declared, "utf-8", "cp1252", "latin1"]):
        try:
            gdf = gpd.read_file(path, encoding=enc)
            print(f"  {enc:>8}: {gdf['name'].head(3).to_list()}")
        except UnicodeDecodeError as exc:
            print(f"  {enc:>8}: failed — {str(exc)[:60]}")

orte_utf8.shp: declared encoding <none>
     utf-8: ['Fürth', 'Münster', 'Düsseldorf']
    cp1252: ['FÃ¼rth', 'MÃ¼nster', 'DÃ¼sseldorf']
    latin1: ['FÃ¼rth', 'MÃ¼nster', 'DÃ¼sseldorf']
orte_cp1252.shp: declared encoding <none>
     utf-8: failed — 'utf-8' codec can't decode byte 0xfc in position 1: invalid 
    cp1252: ['Fürth', 'Münster', 'Düsseldorf']
    latin1: ['Fürth', 'Münster', 'Düsseldorf']

Printing the first few values under each candidate is the fastest reliable test: Fürth is right, FÃ¼rth is UTF-8 bytes read as a single-byte code page, and an exception means the bytes are not valid in the codec you tried.

How mojibake happens

The bytes are fine — the mismatch is between the encoder and the decoder.

Find the declared encoding, test candidates, then fix

Detection is a short ladder — and the last rung is always a human looking at the text.

Read the `.cpg` and the DBF header

The .cpg file holds one line of text naming a code page. Common values are UTF-8, ISO-8859-1, CP1252, LDID/87. GDAL writes one for every shapefile, which is what the untouched Natural Earth file and a fresh write both show:

for shp in (NE, Path("data/raw/fresh_cp1252.shp")):
    stem = shp.name.replace(".shp", "")
    folder = "/vsizip/data/ne_places.zip" if shp == NE else str(shp.parent)
    cpg = gdal.VSIFOpenL(f"{folder}/{stem}.cpg", "rb")
    text = gdal.VSIFReadL(1, 64, cpg).decode() if cpg else None
    if cpg:
        gdal.VSIFCloseL(cpg)
    dbf = gdal.VSIFOpenL(f"{folder}/{stem}.dbf", "rb")
    header = gdal.VSIFReadL(1, 32, dbf)
    gdal.VSIFCloseL(dbf)
    print(f"{stem:26} .cpg: {text!r:12} language driver id (byte 29): {header[29]}")

ne_10m_populated_places    .cpg: 'UTF-8'      language driver id (byte 29): 0
fresh_cp1252               .cpg: 'CP1252'     language driver id (byte 29): 0

Both carry a .cpg, and neither sets the language driver byte: an id of 0 means the header says nothing, which is the usual case. Lose the .cpg and nothing in the file records the encoding at all.

Know what the reader does without one

With no .cpg and no language driver id, GDAL decodes the .dbf as ISO-8859-1. That default explains both failures in the problem statement:

for written in ("utf8", "cp1252"):
    default = gpd.read_file(f"data/raw/orte_{written}.shp")["name"].iloc[0]
    print(f"written as {written:6}, read with no encoding=: {default!r}")

written as utf8  , read with no encoding=: 'FÃ¼rth'
written as cp1252, read with no encoding=: 'Fürth'

The Windows-1252 file happens to read correctly, because Windows-1252 and ISO-8859-1 agree on every letter in these names. The UTF-8 file does not: each ü is two bytes, and ISO-8859-1 turns them into two characters, Ã¼.

Try candidates and look at the output

Automatic detection is a heuristic; your eyes are the ground truth. Polish and Russian names from the same Natural Earth file, written as Windows-1250 and Windows-1251 without a .cpg, show why:

CANDIDATES = ["utf-8", "cp1252", "latin1", "cp1250", "cp1251"]

def preview(path, column="name", candidates=CANDIDATES, n=3):
    for enc in candidates:
        try:
            values = gpd.read_file(path, encoding=enc, rows=n)[column].to_list()
            print(f"  {enc:>7}: {values}")
        except UnicodeDecodeError:
            print(f"  {enc:>7}: UnicodeDecodeError")

for name in ("miasta_cp1250.shp", "goroda_cp1251.shp"):
    print(name)
    preview(f"data/raw/{name}")

miasta_cp1250.shp
    utf-8: UnicodeDecodeError
   cp1252: ['Elbl¹g', 'Inowroc³aw', 'E³k']
   latin1: ['Elbl¹g', 'Inowroc³aw', 'E³k']
   cp1250: ['Elbląg', 'Inowrocław', 'Ełk']
   cp1251: ['Elbl№g', 'Inowrocіaw', 'Eіk']
goroda_cp1251.shp
    utf-8: UnicodeDecodeError
   cp1252: ['Íàçðàíü', 'Óñòü-Îðäûíñêèé', 'Ìàéêîï']
   latin1: ['Íàçðàíü', 'Óñòü-Îðäûíñêèé', 'Ìàéêîï']
   cp1250: ['Íŕçđŕíü', 'Óńňü-Îđäűíńęčé', 'Ěŕéęîď']
   cp1251: ['Назрань', 'Усть-Ордынский', 'Майкоп']

cp1252 and latin1 returned something for both files, without an error, and it is wrong. Only the right code page gives readable Polish or Russian — and you can only tell which one that is by reading the result.

The preview above, for one name from each file.

Use a detector as a hint, not an answer

import charset_normalizer

for name in ("orte_cp1252", "miasta_cp1250", "goroda_cp1251", "orte_utf8"):
    raw = Path(f"data/raw/{name}.dbf").read_bytes()
    best = charset_normalizer.from_bytes(raw).best()
    print(f"{name:14} guess: {best.encoding if best else None}")

orte_cp1252    guess: None
miasta_cp1250  guess: None
goroda_cp1251  guess: None
orte_utf8      guess: utf_8

charset_normalizer recognised the UTF-8 file and gave no answer at all for the three code-page files. Detectors work on statistics and are weakest exactly where it matters: short strings, many proper nouns, and encodings that differ in only a handful of bytes — and a .dbf is mostly padding spaces and numbers around a few names. Treat any answer as a candidate to verify.

Fix mojibake that is already in your data

If a previous step decoded with the wrong codec and saved the result, the damage is in the text rather than the bytes. It is often reversible:

def unmojibake(s: str) -> str:
    """Reverse a UTF-8-read-as-Latin-1 round trip."""
    try:
        return s.encode("latin1").decode("utf-8")
    except (UnicodeEncodeError, UnicodeDecodeError):
        return s

garbled = gpd.read_file("data/raw/orte_utf8.shp")["name"]
print([unmojibake(s) for s in garbled.head(3)])
print("ftfy:", [ftfy.fix_text(s) for s in garbled.head(3)])
print("already clean, left alone:", unmojibake("Gießen"))

['Fürth', 'Münster', 'Düsseldorf']
ftfy: ['Fürth', 'Münster', 'Düsseldorf']
already clean, left alone: Gießen

The re-encoding codec must be the one that did the damage — here Latin-1, GDAL's default. The ftfy package works that out itself and handles double and triple round trips. Repair once, verify, and store the result as UTF-8 so it cannot happen again.

Set the encoding when writing, too

Writing a shapefile without saying what encoding you used passes the problem downstream. GDAL writes the .cpg for you — the trouble starts when it gets separated from the .dbf:

places.to_file("data/out/orte.shp", encoding="utf-8")
print("orte.cpg:", Path("data/out/orte.cpg").read_text())
places.to_file("data/out/orte.gpkg", layer="orte")
print("GeoPackage:", gpd.read_file("data/out/orte.gpkg")["name"].head(3).to_list())

orte.cpg: UTF-8
GeoPackage: ['Fürth', 'Münster', 'Düsseldorf']

GeoPackage, GeoJSON (UTF-8 by specification) and Parquet all remove the question permanently.

Override a wrong declaration

A .cpg can lie. Here a Windows-1252 file carries a .cpg that says UTF-8:

shutil.copy("data/raw/orte_cp1252.dbf", "data/raw/liar.dbf")
for ext in (".shp", ".shx"):
    shutil.copy(f"data/raw/orte_cp1252{ext}", f"data/raw/liar{ext}")
Path("data/raw/liar.cpg").write_text("UTF-8")
try:
    print(gpd.read_file("data/raw/liar.shp")["name"].head(3).to_list())
except UnicodeDecodeError as exc:
    print("trusting the .cpg:", type(exc).__name__)
print("encoding='cp1252':", gpd.read_file("data/raw/liar.shp", encoding="cp1252")["name"].head(3).to_list())

trusting the .cpg: UnicodeDecodeError
encoding='cp1252': ['Fürth', 'Münster', 'Düsseldorf']

An explicit encoding= wins over the .cpg. GDAL's own SHAPE_ENCODING configuration option does the same job for other GDAL tools; from GeoPandas, the argument is simpler and scoped to one read.

Normalise the text once you have it right

composed = "Düsseldorf"
decomposed = unicodedata.normalize("NFD", composed)          # how macOS often stores it
print(composed == decomposed, len(composed), len(decomposed))
print(unicodedata.normalize("NFC", decomposed) == composed)

False 10 11
True

The two strings look identical and are not equal: the decomposed form stores ü as u plus a combining diaeresis, one character longer. Two visually identical names that fail an equality test are almost always a normalisation difference — a joining bug waiting to happen. Normalise text keys to NFC before joining.

Recovering German, Polish and Russian place names

The data used on this page

Natural Earth's 1:10m populated places (public domain), which carry names in many languages in UTF-8. The setup writes the German, Polish and Russian names of a few cities to shapefiles in the code pages a desktop GIS in each country would use, then deletes each .cpg — reproducing a delivery that lost its sidecar.

import shutil
import unicodedata
from pathlib import Path

import ftfy
import geopandas as gpd
import requests
from osgeo import gdal

for folder in ("data/raw", "data/out", "data/clean"):
    Path(folder).mkdir(parents=True, exist_ok=True)
if not Path("data/ne_places.zip").exists():
    Path("data/ne_places.zip").write_bytes(requests.get(
        "https://naciscdn.org/naturalearth/10m/cultural/ne_10m_populated_places.zip", timeout=120).content)
NE = Path("ne_10m_populated_places.shp")
world = gpd.read_file("/vsizip/data/ne_places.zip/ne_10m_populated_places.shp",
                      columns=["NAME", "NAME_DE", "NAME_PL", "NAME_RU", "ADM0NAME"])

def pick(country, column, pattern, n=5):
    rows = world[(world["ADM0NAME"] == country) & world[column].str.contains(pattern)]
    return rows[[column, "geometry"]].rename(columns={column: "name"}).head(n).reset_index(drop=True)

places = pick("Germany", "NAME_DE", "[äöüß]")
sets = {"orte_utf8": (places, "utf-8"), "orte_cp1252": (places, "cp1252"),
        "miasta_cp1250": (pick("Poland", "NAME_PL", "[ąćęłńóśźż]"), "cp1250"),
        "goroda_cp1251": (pick("Russia", "NAME_RU", "."), "cp1251")}
for stem, (frame, encoding) in sets.items():
    frame.to_file(f"data/raw/{stem}.shp", encoding=encoding)
    Path(f"data/raw/{stem}.cpg").unlink()                     # the sidecar that went missing
places.to_file("data/raw/fresh_cp1252.shp", encoding="cp1252")  # this one keeps its .cpg
print({stem: frame["name"].head(2).to_list() for stem, (frame, _) in sets.items()})

{'orte_utf8': ['Fürth', 'Münster'], 'orte_cp1252': ['Fürth', 'Münster'], 'miasta_cp1250': ['Elbląg', 'Inowrocław'], 'goroda_cp1251': ['Назрань', 'Усть-Ордынский']}

Example 1: a reader that resolves encoding automatically

FALLBACKS = ["utf-8", "cp1252", "latin1"]
MARKERS = ("Ã", "Â", "�")

def read_shapefile(path, text_column=None, extra=()):
    path = Path(path)
    cpg = path.with_suffix(".cpg")
    declared = cpg.read_text(encoding="ascii", errors="ignore").strip() if cpg.exists() else None

    ordered = []
    for c in [declared, *extra, *FALLBACKS]:
        if c and c.lower().replace("-", "").replace("_", "") not in {o.lower().replace("-", "").replace("_", "") for o in ordered}:
            ordered.append(c)

    last_error = None
    for enc in ordered:
        try:
            gdf = gpd.read_file(path, encoding=enc)
        except (UnicodeDecodeError, LookupError) as exc:
            last_error = exc
            continue
        if text_column and text_column in gdf.columns:
            sample = " ".join(gdf[text_column].dropna().astype(str).head(50))
            if any(marker in sample for marker in MARKERS):
                last_error = ValueError(f"{enc} produced mojibake")
                continue
        gdf.attrs["encoding"] = enc
        return gdf
    raise RuntimeError(f"could not decode {path} with any of {ordered}: {last_error}")

for stem in ("orte_utf8", "orte_cp1252", "miasta_cp1250"):
    gdf = read_shapefile(f"data/raw/{stem}.shp", "name", extra=["cp1250"] if "cp1250" in stem else ())
    print(f"{stem:14} -> {gdf.attrs['encoding']:7} {gdf['name'].head(2).to_list()}")

orte_utf8      -> utf-8   ['Fürth', 'Münster']
orte_cp1252    -> cp1252  ['Fürth', 'Münster']
miasta_cp1250  -> cp1250  ['Elbląg', 'Inowrocław']

Scanning for the Ã/Â/� markers catches the case where a codec succeeds but produces nonsense. It cannot catch the Polish file read as cp1252, which produces wrong letters but no markers — which is why the regional code page is passed in extra, from knowing where the data came from.

Example 2: an encoding audit across a folder

def audit(folder="data/raw"):
    rows = []
    for shp in sorted(Path(folder).glob("*.shp")):
        cpg = shp.with_suffix(".cpg")
        declared = cpg.read_text(errors="ignore").strip() if cpg.exists() else ""
        try:
            sample = " ".join(gpd.read_file(shp, rows=100)["name"].dropna().astype(str))
            status = "SUSPECT" if any(m in sample for m in MARKERS) else "ok?" if not declared else "ok"
        except UnicodeDecodeError:
            status = "DECODE ERROR"
        rows.append((status, declared or "-", shp.name))
    return rows

for status, declared, name in audit():
    print(f"{status:<13} {declared:<7} {name}")

ok            CP1252  fresh_cp1252.shp
SUSPECT       -       goroda_cp1251.shp
DECODE ERROR  UTF-8   liar.shp
ok?           -       miasta_cp1250.shp
ok?           -       orte_cp1252.shp
SUSPECT       -       orte_utf8.shp

ok? marks a file with no declaration that read without markers. The Polish file lands there with wrong letters (Elbl¹g), and only luck put the Russian file under SUSPECT — so a missing .cpg should always send someone to look. The file with the lying .cpg fails to decode, which is the best outcome of the lot. Running this over a delivery before processing turns a subtle data-quality problem into a checklist.

Example 2’s output beside what each file really holds.

Example 3: convert to UTF-8 GeoPackage at the boundary

for stem, encoding in (("orte_utf8", "utf-8"), ("orte_cp1252", "cp1252"),
                       ("miasta_cp1250", "cp1250"), ("goroda_cp1251", "cp1251")):
    gdf = gpd.read_file(f"data/raw/{stem}.shp", encoding=encoding)
    gdf["name"] = gdf["name"].map(lambda s: unicodedata.normalize("NFC", s))
    gdf.to_file(f"data/clean/{stem}.gpkg", driver="GPKG")
    print(f"{stem:14} [{encoding}] -> {gpd.read_file(f'data/clean/{stem}.gpkg')['name'].iloc[0]}")

orte_utf8      [utf-8] -> Fürth
orte_cp1252    [cp1252] -> Fürth
miasta_cp1250  [cp1250] -> Elbląg
goroda_cp1251  [cp1251] -> Назрань

One conversion at the boundary of your pipeline, with the encoding decided by a person once, and every downstream step gets UTF-8 for free.

Example 4: catch encoding problems in a test

import re

MOJIBAKE = re.compile(r"[ÃÂ][\x80-\xbf]|�")

def mis_encoded(path):
    names = gpd.read_file(path)["name"].astype(str)
    return names[names.str.contains(MOJIBAKE, na=False)].to_list()

print("clean GeoPackage:", mis_encoded("data/clean/orte_utf8.gpkg"))
print("raw UTF-8 shapefile read with defaults:", mis_encoded("data/raw/orte_utf8.shp"))

clean GeoPackage: []
raw UTF-8 shapefile read with defaults: ['FÃ¼rth', 'MÃ¼nster', 'DÃ¼sseldorf', 'GieÃ\x9fen', 'OsnabrÃ¼ck']

Bytes, encodings and mojibake

Text on disk is bytes. An encoding is the agreement about which bytes mean which characters, and decoding with the wrong agreement produces the wrong characters — or, when the byte sequence is not legal in the assumed encoding, a UnicodeDecodeError. UTF-8 is strict about legal sequences, which is why it errors; the single-byte code pages accept almost anything, which is why they silently produce mojibake instead.

Six symptoms, six fixes — and only one of them needs a detector.

The shapefile format makes this worse than it needs to be. Its attribute table is dBase, a format from the early 1980s, when the code page was a property of the machine rather than the file. ESRI later added the .cpg sidecar and a language-driver byte in the header, but neither is required, GDAL does not set the byte, and files travel without their sidecars constantly — a .zip that contains only .shp, .shx and .dbf has already lost both the CRS and the encoding.

That is why the practical procedure is a short ladder rather than a lookup. Read the declaration if it exists; try UTF-8, because modern exports usually are; then try the code page appropriate to the data's region. Then look at the text, because Latin-1 decodes any byte sequence without complaint and will happily give you FÃ¼rth forever. Detection libraries help narrow the list, but on short proper nouns they are frequently wrong.

Once you have readable text, convert and stop the problem at your pipeline's boundary. GeoPackage is UTF-8 by specification, GeoJSON is UTF-8 by specification, and Parquet stores strings as UTF-8. Any of them removes an entire class of failure from every downstream step — which is a better use of effort than teaching each step to guess.

Readers that never fail, and labels that lie

latin1 never raises: It maps all 256 byte values to characters, and it is GDAL's default when a shapefile declares nothing. Absence of an exception is not evidence of a correct decode.
A .cpg can be wrong: Writers sometimes stamp UTF-8 regardless. Pass encoding= to read_file, which overrides it.
Field names are limited to 10 bytes: With multi-byte characters, truncation can split a character. Rename columns to ASCII before writing shapefiles.
Double encoding is reversible, triple usually is not: s.encode("latin1").decode("utf-8") fixes one round trip; ftfy handles more, but data loss becomes likely.
NFC vs NFD: macOS often stores decomposed forms. Normalise to NFC before joining on text keys, or equal-looking values will not match.
The .dbf is not the only text: Layer and field names in a GeoPackage are UTF-8, but a shapefile's field names carry the same encoding ambiguity as its values.
CSV has the same problem: pd.read_csv(..., encoding=...) follows exactly this ladder, plus a BOM check for utf-8-sig.

Internal links

FAQ

What encoding do shapefiles use?

There is no single answer — the format does not require one. Check for a .cpg sidecar, then try UTF-8 and the regional code page (cp1252 in Western Europe, cp1250 in Central Europe, cp1251 for Cyrillic).

Why does `latin1` never fail?

Because it maps every possible byte to a character, so no sequence is illegal. That makes it a useful last resort for getting data in, but it will silently produce wrong characters where UTF-8 would have told you.

How do I fix names that are already garbled, like `MÃ¼nchen`?

Re-encode with the codec that did the damage and decode as UTF-8 — s.encode("latin1").decode("utf-8") when GDAL's Latin-1 default read UTF-8 bytes. The ftfy package works out the codec itself and handles more complex cases.

Should I trust `chardet` or `charset_normalizer`?

As a hint. They are statistical and least reliable on short strings and proper nouns — exactly what attribute tables contain. Always confirm by looking at the decoded values.

How do I stop this happening again?

Convert to GeoPackage, GeoJSON or Parquet at the point of ingest. All three are UTF-8 by specification, so no downstream step has to guess.

Why do two identical-looking names fail to match in a join?

Unicode normalisation: one may be composed (NFC) and the other decomposed (NFD). Normalise both with unicodedata.normalize("NFC", s) before joining.

Does the `.cpg` affect the geometry?

No. Geometry lives in the .shp as binary coordinates and is unaffected by encoding. Only the attribute table and field names are involved.