🔧 Fix Files & formats Data cleaning GIS Data Cleaning · step 30 of 33 Updated

Garbled Attribute Text and UnicodeDecodeError from a Shapefile: How to Fix Encoding

Problem statement

The place names came back wrong:

gdf = gpd.read_file("data/raw/gemeinden.shp")
print(gdf["name"].head(3).to_list())
# ['München', 'Nürnberg', 'Würzburg']

Or the read fails outright:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 3:
invalid start byte

A shapefile stores attributes in a dBase (.dbf) table, which has no reliable way to say what character encoding it used. There is an optional .cpg sidecar and a one-byte language driver code in the header, and both are frequently missing or wrong. The reader then has to guess, and a wrong guess produces either mojibake — ü where ü belongs — or a hard decode error.

Common causes:

  • no .cpg file, so the driver falls back to UTF-8 or to a system default
  • the data was written as Windows-1252 or ISO-8859-1 by a desktop GIS
  • Central and Eastern European data written as Windows-1250, Cyrillic as Windows-1251
  • Chinese, Japanese or Korean data in GBK, Shift-JIS or EUC-KR
  • a file that has already been double-encoded by an earlier faulty conversion
  • the .cpg says UTF-8 but the bytes are not

Quick answer

To read a shapefile with the correct encoding:

  1. look for a .cpg file next to the .shp and read it
  2. try UTF-8 first; if it fails or looks wrong, try cp1252, then latin1
  3. pass the encoding explicitly: gpd.read_file(path, encoding="cp1252")
  4. verify by looking at the characters, not by the absence of an exception
  5. write outputs as UTF-8 — GeoPackage always is — and stop the problem spreading
from pathlib import Path
import geopandas as gpd

path = Path("data/raw/gemeinden.shp")

cpg = path.with_suffix(".cpg")
declared = cpg.read_text(encoding="ascii", errors="ignore").strip() if cpg.exists() else None
print("declared encoding:", declared or "<none>")

for enc in filter(None, [declared, "utf-8", "cp1252", "latin1"]):
    try:
        gdf = gpd.read_file(path, encoding=enc)
        print(f"{enc:>8}: {gdf['name'].head(3).to_list()}")
    except UnicodeDecodeError as exc:
        print(f"{enc:>8}: failed — {exc}")

Printing the first few values under each candidate is the fastest reliable test: München is right, München is UTF-8 bytes read as cp1252, and M?nchen is a lossy fallback.

How mojibake happens

The path from source text through encoding bytes to a wrong decode producing mojibake.
The bytes are fine — the mismatch is between the encoder and the decoder.

Step-by-step solution

Vertical steps: check cpg, try utf-8, try cp1252, verify visually, convert and write UTF-8.
Detection is a short ladder — and the last rung is always a human looking at the text.

Read the .cpg and the DBF header

The .cpg file holds one line of text naming a code page. Common values are UTF-8, ISO-8859-1, CP1252, LDID/87.

from pathlib import Path

shp = Path("data/raw/gemeinden.shp")
for ext in (".cpg", ".prj", ".dbf"):
    p = shp.with_suffix(ext)
    print(f"{ext}: {'present' if p.exists() else 'MISSING'}")

cpg = shp.with_suffix(".cpg")
if cpg.exists():
    print("cpg says:", cpg.read_bytes()[:64])

The DBF header's byte 29 is the language driver id, which some writers set and most do not:

with open(shp.with_suffix(".dbf"), "rb") as fh:
    header = fh.read(32)
print("language driver id:", header[29])   # 0x00 = unset, 0x03 = cp1252, 0x57 = ANSI …

An id of 0 means the file says nothing at all, which is the usual case.

Try candidates and look at the output

Automatic detection is a heuristic; your eyes are the ground truth.

import geopandas as gpd

CANDIDATES = ["utf-8", "cp1252", "latin1", "cp1250", "cp1251", "gbk", "shift_jis"]

def preview(path, column, candidates=CANDIDATES, n=5):
    for enc in candidates:
        try:
            values = gpd.read_file(path, encoding=enc, rows=n)[column].to_list()
            print(f"{enc:>10}: {values}")
        except UnicodeDecodeError:
            print(f"{enc:>10}: UnicodeDecodeError")
        except Exception as exc:
            print(f"{enc:>10}: {type(exc).__name__}: {exc}")

preview("data/raw/gemeinden.shp", "name")

Note that latin1 never raises — it maps every byte to a character — so "no exception" proves nothing. Only the rendered text does.

Use a detector as a hint, not an answer 

from pathlib import Path
import charset_normalizer          # or: chardet

raw = Path("data/raw/gemeinden.dbf").read_bytes()
result = charset_normalizer.from_bytes(raw).best()
print("guess:", result.encoding, "confidence:", result.coherence)

Detectors work on statistics and are weakest exactly where it matters: short strings, many proper nouns, and encodings that differ in only a handful of bytes. Treat the answer as a candidate to verify.

Fix mojibake that is already in your data

If a previous step decoded with the wrong codec and saved the result, the damage is in the text rather than the bytes. It is often reversible.

def unmojibake(s: str) -> str:
    """Reverse a UTF-8-read-as-cp1252 round trip."""
    try:
        return s.encode("cp1252").decode("utf-8")
    except (UnicodeEncodeError, UnicodeDecodeError):
        return s

print(unmojibake("München"))       # München
gdf["name"] = gdf["name"].map(unmojibake)

The ftfy package generalises this and handles double and triple round trips. Repair once, verify, and store the result as UTF-8 so it cannot happen again.

Set the encoding when writing, too

Writing a shapefile without saying what encoding you used passes the problem downstream.

# shapefile: write UTF-8 and say so
gdf.to_file("data/out/gemeinden.shp", driver="ESRI Shapefile", encoding="utf-8")
# GeoPandas writes a .cpg alongside; confirm it exists

Better: stop writing shapefiles.

# GeoPackage is UTF-8 by definition — no sidecar, no ambiguity
gdf.to_file("data/out/gemeinden.gpkg", layer="gemeinden", driver="GPKG")

GeoPackage, GeoJSON (UTF-8 by specification) and Parquet all remove the question permanently.

Control the driver's behaviour when you must use shapefiles

GDAL's shapefile driver has two options worth knowing.

import os
os.environ["SHAPE_ENCODING"] = ""        # ignore the .cpg / LDID and use the raw bytes
os.environ["SHAPE_RESTORE_SHX"] = "YES"  # unrelated, but rebuilds a missing .shx

Setting SHAPE_ENCODING to an empty string tells the driver not to recode at all, which is useful when the .cpg is actively wrong — you then decode in Python yourself.

Normalise the text once you have it right 

import unicodedata

gdf["name"] = (
    gdf["name"]
    .astype("string")
    .map(lambda s: unicodedata.normalize("NFC", s) if s is not None else s)
    .str.strip()
)

NFC normalisation matters when data comes from macOS, where ü is often stored as u plus a combining diaeresis. Two visually identical strings that fail an equality test are almost always a normalisation difference — a joining bug waiting to happen.

Code examples

Example 1: a reader that resolves encoding automatically 

from pathlib import Path
import geopandas as gpd

FALLBACKS = ["utf-8", "cp1252", "latin1"]

def read_shapefile(path, text_column=None, extra=()):
    path = Path(path)
    cpg = path.with_suffix(".cpg")
    declared = cpg.read_text(encoding="ascii", errors="ignore").strip() if cpg.exists() else None

    candidates = [c for c in [declared, *extra, *FALLBACKS] if c]
    seen, ordered = set(), []
    for c in candidates:
        key = c.lower().replace("-", "").replace("_", "")
        if key not in seen:
            seen.add(key)
            ordered.append(c)

    last_error = None
    for enc in ordered:
        try:
            gdf = gpd.read_file(path, encoding=enc)
        except (UnicodeDecodeError, LookupError) as exc:
            last_error = exc
            continue
        if text_column and text_column in gdf.columns:
            sample = " ".join(gdf[text_column].dropna().astype(str).head(50))
            if any(marker in sample for marker in ("Ã", "Â", "�")):
                last_error = ValueError(f"{enc} produced mojibake")
                continue
        gdf.attrs["encoding"] = enc
        return gdf

    raise RuntimeError(f"could not decode {path} with any of {ordered}: {last_error}")

Scanning for the Ã/Â/� markers catches the case where a codec succeeds but produces nonsense.

Example 2: an encoding audit across a folder 

from pathlib import Path
import geopandas as gpd

def audit(folder="data/raw"):
    rows = []
    for shp in sorted(Path(folder).rglob("*.shp")):
        cpg = shp.with_suffix(".cpg")
        declared = cpg.read_text(errors="ignore").strip() if cpg.exists() else ""
        try:
            gdf = gpd.read_file(shp, rows=100)
            text_cols = [c for c in gdf.columns if gdf[c].dtype == "object" and c != "geometry"]
            sample = " ".join(gdf[c].dropna().astype(str).head(20).sum() for c in text_cols[:3])
            suspicious = any(m in sample for m in ("Ã", "Â", "�"))
            rows.append((shp.name, declared or "-", "SUSPECT" if suspicious else "ok"))
        except UnicodeDecodeError:
            rows.append((shp.name, declared or "-", "DECODE ERROR"))
    return rows

for name, declared, status in audit():
    print(f"{status:<13} {declared:<10} {name}")

Running this over a delivery before processing turns a subtle data-quality problem into a checklist.

Example 3: convert a whole folder to UTF-8 GeoPackage 

from pathlib import Path
import geopandas as gpd

SRC, OUT = Path("data/raw"), Path("data/clean")
OUT.mkdir(parents=True, exist_ok=True)

for shp in sorted(SRC.rglob("*.shp")):
    gdf = read_shapefile(shp)                       # from Example 1
    dest = (OUT / shp.relative_to(SRC)).with_suffix(".gpkg")
    dest.parent.mkdir(parents=True, exist_ok=True)
    gdf.to_file(dest, driver="GPKG")
    print(f"{shp.name} [{gdf.attrs['encoding']}] → {dest.relative_to(OUT)}")

One conversion at the boundary of your pipeline, and every downstream step gets UTF-8 for free.

Example 4: catch encoding problems in a test 

import re
import geopandas as gpd

MOJIBAKE = re.compile(r"[ÃÂ]\s*[\x80-\xbf]|�")

def test_no_mojibake_in_place_names():
    gdf = gpd.read_file("data/clean/gemeinden.gpkg")
    bad = gdf[gdf["name"].astype(str).str.contains(MOJIBAKE, na=False)]
    assert bad.empty, f"{len(bad)} names look mis-encoded: {bad['name'].head().to_list()}"

Explanation

Text on disk is bytes. An encoding is the agreement about which bytes mean which characters, and decoding with the wrong agreement produces the wrong characters — or, when the byte sequence is not legal in the assumed encoding, a UnicodeDecodeError. UTF-8 is strict about legal sequences, which is why it errors; the single-byte code pages accept anything, which is why they silently produce mojibake instead.

Triage table mapping encoding symptoms to causes and fixes.
Six symptoms, six fixes — and only one of them needs a detector.

The shapefile format makes this worse than it needs to be. Its attribute table is dBase, a format from the early 1980s, when the code page was a property of the machine rather than the file. ESRI later added the .cpg sidecar and a language-driver byte in the header, but neither is required, and files travel without their sidecars constantly — a .zip that contains only .shp, .shx and .dbf has already lost both the CRS and the encoding.

That is why the practical procedure is a short ladder rather than a lookup. Read the declaration if it exists; try UTF-8, because modern exports usually are; then try the code page appropriate to the data's region. Then look at the text, because latin1 decodes any byte sequence without complaint and will happily give you München forever. Detection libraries help narrow the list, but on short proper nouns they are frequently wrong.

Once you have readable text, convert and stop the problem at your pipeline's boundary. GeoPackage is UTF-8 by specification, GeoJSON is UTF-8 by specification, and Parquet stores strings as UTF-8. Any of them removes an entire class of failure from every downstream step — which is a better use of effort than teaching each step to guess.

Edge cases or notes

  • latin1 never raises: It maps all 256 byte values to characters. Absence of an exception is not evidence of a correct decode.
  • A .cpg can be wrong: Writers sometimes stamp UTF-8 regardless. Set SHAPE_ENCODING="" to bypass the declaration and decode yourself.
  • Field names are limited to 10 bytes: With multi-byte characters, truncation can split a character. Rename columns to ASCII before writing shapefiles.
  • Double encoding is reversible, triple usually is not: s.encode("cp1252").decode("utf-8") fixes one round trip; ftfy handles more, but data loss becomes likely.
  • NFC vs NFD: macOS often stores decomposed forms. Normalise to NFC before joining on text keys, or equal-looking values will not match.
  • The .dbf is not the only text: Layer and field names in a GeoPackage are UTF-8, but a shapefile's field names carry the same encoding ambiguity as its values.
  • CSV has the same problem: pd.read_csv(..., encoding=...) follows exactly this ladder, plus a BOM check for utf-8-sig.

FAQ

What encoding do shapefiles use?

There is no single answer — the format does not require one. Check for a .cpg sidecar, then try UTF-8 and the regional code page (cp1252 in Western Europe, cp1250 in Central Europe, cp1251 for Cyrillic).

Why does latin1 never fail?

Because it maps every possible byte to a character, so no sequence is illegal. That makes it a useful last resort for getting data in, but it will silently produce wrong characters where UTF-8 would have told you.

How do I fix names that are already garbled, like München?

Re-encode and re-decode: s.encode("cp1252").decode("utf-8"). That reverses the common UTF-8-read-as-cp1252 round trip. The ftfy package handles more complex cases.

Should I trust chardet or charset_normalizer?

As a hint. They are statistical and least reliable on short strings and proper nouns — exactly what attribute tables contain. Always confirm by looking at the decoded values.

How do I stop this happening again?

Convert to GeoPackage, GeoJSON or Parquet at the point of ingest. All three are UTF-8 by specification, so no downstream step has to guess.

Why do two identical-looking names fail to match in a join?

Unicode normalisation: one may be composed (NFC) and the other decomposed (NFD). Normalise both with unicodedata.normalize("NFC", s) before joining.

Does the .cpg affect the geometry?

No. Geometry lives in the .shp as binary coordinates and is unaffected by encoding. Only the attribute table and field names are involved.