Fixing an Uploaded File That Breaks a Map App

Problem statement

The app accepts a file upload, and users send everything: a zipped shapefile missing its .prj, a CSV with coordinates in three different formats, a 400 MB GeoPackage, a GeoJSON in a national grid, and a spreadsheet renamed to .csv.

Each produces a different failure and most of them produce the same symptom — a stack trace in the browser, or a blank map with no explanation:

portugal_excel_export.csv       UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 1
portugal_workbook_renamed.csv   UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6 in position 1

Those two, from gpd.read_file on real uploads made for this page (Example 1), are the lucky ones. Three more read without complaint and wrong: a zipped shapefile without its .prj comes back with crs None; a GeoJSON written in Portugal's national grid comes back as EPSG:4326 with coordinates in the hundreds of thousands; and a GeoPackage with two layers silently gives the first.

An upload is untrusted input. The fix is a validation pipeline that checks the file before anything tries to render it, and reports what is wrong in terms the user can act on.

Quick answer

Validate in stages, and fail with a message rather than a trace:

import streamlit as st

MAX_UPLOAD_MB = 50
SUPPORTED = {".geojson", ".json", ".gpkg", ".zip", ".csv", ".parquet"}

upload = st.file_uploader("Upload a layer",
                          type=[e.lstrip(".") for e in SUPPORTED])

if upload is not None:
    problems = validate_upload(upload)
    if problems:
        st.error("This file cannot be used:")
        for problem in problems:
            st.write(f"- {problem}")
        st.stop()

    gdf = read_upload(upload)
    st.success(f"Loaded {len(gdf):,} features in {gdf.crs.to_string()}")

Also set the platform limit, because the default is generous:

streamlit run app.py --server.maxUploadSize=50

Bound size, sniff type, unzip properly, establish the CRS

1. Bound the size before reading anything

Streamlit's default upload limit is 200 MB, and a GeoJSON takes several times its file size in memory once read — measured, a 27.0 MB GeoJSON of Europe's NUTS 3 regions grew the reading process by 65 MB, 2.4 times the file (Example 1).

size_mb = upload.size / 1e6
if size_mb > MAX_UPLOAD_MB:
    st.error(f"{size_mb:.0f} MB exceeds the {MAX_UPLOAD_MB} MB limit. "
             f"Upload a filtered extract, or use the bulk import.")
    st.stop()

Set --server.maxUploadSize as well, so the transfer is refused rather than completed and then rejected.

2. Check the type by content, not by extension

An extension is a claim. Read the first bytes:

MAGIC = {b"PK\x03\x04": "zip", b"SQLite format 3\x00": "gpkg",
         b"PAR1": "parquet"}


def sniff(head: bytes) -> str:
    for signature, kind in MAGIC.items():
        if head.startswith(signature):
            return kind
    text = head.lstrip()[:1]
    if text in (b"{", b"["):
        return "json"
    return "text"

A .geojson that is actually a zip, or a .csv that is an Excel workbook, is common enough to be worth two lines.

3. Handle a zipped shapefile properly

A shapefile is several files, so it arrives zipped — and frequently incomplete:

import zipfile, io, tempfile, os


def extract_shapefile(upload) -> tuple[str | None, list[str]]:
    problems = []
    with zipfile.ZipFile(io.BytesIO(upload.getvalue())) as archive:
        names = archive.namelist()
        shp = [n for n in names if n.lower().endswith(".shp")]
        if not shp:
            if any(n.startswith("xl/") for n in names):       # a workbook is a zip too
                return None, ["this is an Excel workbook, not a CSV — "
                              "save it as CSV (UTF-8) and upload that"]
            return None, ["the zip contains no .shp file"]
        if len(shp) > 1:
            problems.append(f"{len(shp)} shapefiles in the zip; using the first")

        stem = shp[0][:-4]
        for suffix, consequence in ((".shx", "required"),
                                    (".dbf", "attributes will be missing"),
                                    (".prj", "the CRS will be unknown")):
            if stem + suffix not in names:
                problems.append(f"{suffix} is missing — {consequence}")

        directory = tempfile.mkdtemp()
        archive.extractall(directory)
        return os.path.join(directory, shp[0]), problems

The .prj warning is the important one: without it the file reads and its coordinates mean nothing definite.

4. Establish the CRS, and refuse to guess silently

def looks_like_lonlat(gdf):
    minx, miny, maxx, maxy = gdf.total_bounds
    return -180 <= minx <= maxx <= 180 and -90 <= miny <= maxy <= 90


def resolve_crs(gdf, declared_epsg=None):
    if gdf.crs is not None and not (gdf.crs.is_geographic and not looks_like_lonlat(gdf)):
        return gdf.to_crs(4326), None
    if declared_epsg:
        return gdf.set_crs(declared_epsg, allow_override=True).to_crs(4326), \
            f"assumed EPSG:{declared_epsg} as declared"
    if gdf.crs is None and looks_like_lonlat(gdf):
        return gdf.set_crs(4326), \
            "no CRS declared; the coordinates look like lon/lat, assumed EPSG:4326"
    return None, (f"the coordinates {gdf.total_bounds.round(0).tolist()} are not lon/lat "
                  f"{'although the file says ' + gdf.crs.to_string() if gdf.crs else 'and no CRS is declared'}. "
                  f"Tell the app which EPSG code this is.")

Guessing from the bounds is defensible when it is reported; guessing silently is how a layer ends up plotted in the Gulf of Guinea. The bounds check has to run even when a CRS is declared: GeoJSON is defined to be WGS 84, so a GeoJSON written in a national grid reads as EPSG:4326 with coordinates in the hundreds of thousands (Example 1).

5. Validate the geometry

def geometry_report(gdf):
    report = {
        "features": len(gdf),
        "null": int(gdf.geometry.isna().sum()),
        "empty": int(gdf.geometry.is_empty.sum()),
        "invalid": int((~gdf.geometry.is_valid).sum()),
        "types": gdf.geom_type.value_counts().to_dict(),
    }
    if report["invalid"]:
        gdf = gdf.assign(geometry=gdf.geometry.make_valid())
    return gdf[~gdf.geometry.isna() & ~gdf.geometry.is_empty], report

Report the counts to the user. "1 invalid geometry was repaired" — Wrocławski, in the Polish upload on this page — is information; a silent repair is a surprise later.

6. Handle a CSV's coordinate columns explicitly

LON_NAMES = {"lon", "long", "longitude", "x", "easting"}
LAT_NAMES = {"lat", "latitude", "y", "northing"}


def coordinate_columns(frame):
    lowered = {c.lower(): c for c in frame.columns}
    lon = next((lowered[n] for n in LON_NAMES if n in lowered), None)
    lat = next((lowered[n] for n in LAT_NAMES if n in lowered), None)
    return lon, lat

Offer the guess and let the user correct it with two selectboxes. Automatic detection that cannot be overridden fails on the first file with x and y columns that are eastings and northings.

An upload pipeline tested against broken files

The data used on this page

The "districts" are Europe's NUTS 3 regions — 1,333 of them once joined to data — from Eurostat GISCO at 1:1 million, with GISCO's urban–rural typology as the category, and the share of the population aged 65 and over for 2019 to 2023 from the Eurostat table demo_r_pjanind3 as rate_2019 … rate_2023. Eurostat refuses the default python-requests user agent, hence the header.

import math
import os

import geopandas as gpd
import pandas as pd
import requests

YEARS = [2019, 2020, 2021, 2022, 2023]


def eurostat(table, **filters):
    """A Eurostat table as a Series indexed by every dimension."""
    stat = requests.get(f"https://ec.europa.eu/eurostat/api/dissemination/statistics/1.0/data/{table}",
                        params={"format": "JSON", **filters},
                        headers={"User-Agent": "spatialworkflow.io guide"}, timeout=120).json()
    codes = [sorted(stat["dimension"][d]["category"]["index"], key=stat["dimension"][d]["category"]["index"].get)
             for d in stat["id"]]
    values = [stat["value"].get(str(i)) for i in range(math.prod(stat["size"]))]
    return pd.Series(values, index=pd.MultiIndex.from_product(codes, names=stat["id"]), dtype=float)


if not os.path.exists("districts.gpkg"):
    nuts = gpd.read_file("https://gisco-services.ec.europa.eu/distribution/v2/nuts/geojson/"
                         "NUTS_RG_01M_2021_4326_LEVL_3.geojson")
    older = (eurostat("demo_r_pjanind3", indic_de="PC_Y65_MAX", unit="PC",
                      sinceTimePeriod=YEARS[0], untilTimePeriod=YEARS[-1])
             .droplevel(["freq", "indic_de", "unit"]).unstack("time"))
    older.columns = [f"rate_{year}" for year in older.columns]
    kind = {1: "urban", 2: "intermediate", 3: "rural"}
    (nuts.assign(category=nuts["URBN_TYPE"].map(kind))
         .rename(columns={"NUTS_ID": "code", "CNTR_CODE": "country", "NAME_LATN": "name"})
         [["code", "country", "name", "category", "geometry"]]
         .merge(older, left_on="code", right_index=True)
         .dropna(subset=[f"rate_{YEARS[-1]}"])
         .to_file("districts.gpkg"))

districts = gpd.read_file("districts.gpkg")
print(f"{len(districts):,} NUTS 3 regions in {districts.country.nunique()} countries; "
      f"share aged 65+ in {YEARS[-1]}: {districts[f'rate_{YEARS[-1]}'].min():.1f}–"
      f"{districts[f'rate_{YEARS[-1]}'].max():.1f}%")
print(districts.category.value_counts().to_dict(),
      f"{districts.name.nunique():,} distinct names")

1,333 NUTS 3 regions in 36 countries; share aged 65+ in 2023: 2.7–36.0%
{'intermediate': 619, 'rural': 438, 'urban': 276} 1,332 distinct names

The broken uploads used on this page

Each file is one of the failures from the top of the page, made from the regions above — Portugal's 25, and Poland's 73, which include one invalid polygon from the source: Wrocławski (PL518) has a ring that touches itself.

import glob
import json
import zipfile

portugal = districts[districts["country"] == "PT"]
poland = districts[districts["country"] == "PL"]
os.makedirs("uploads", exist_ok=True)
os.makedirs("staging", exist_ok=True)

portugal.to_file("staging/portugal.shp")
for name, parts in (("portugal_shapefile.zip", ("shp", "shx", "dbf", "prj", "cpg")),
                    ("portugal_no_prj.zip", ("shp", "shx", "dbf", "cpg"))):
    with zipfile.ZipFile(f"uploads/{name}", "w") as archive:
        for part in parts:
            archive.write(f"staging/portugal.{part}", f"portugal.{part}")

national = json.loads(portugal.to_crs(3763).to_json())     # PT-TM06, EPSG:3763
national.pop("crs", None)                                   # as RFC 7946 writers leave it
with open("uploads/portugal_national_grid.geojson", "w") as out:
    json.dump(national, out)

points = portugal.representative_point()
table = portugal.drop(columns="geometry").assign(longitude=points.x.round(5),
                                                 latitude=points.y.round(5))
table.to_csv("uploads/portugal_excel_export.csv", index=False, encoding="cp1252")
table.to_excel("staging/portugal.xlsx", index=False)
os.replace("staging/portugal.xlsx", "uploads/portugal_workbook_renamed.csv")

poland.to_file("uploads/poland_two_layers.gpkg", layer="nuts3")
poland.dissolve().to_file("uploads/poland_two_layers.gpkg", layer="outline")

for path in sorted(glob.glob("uploads/*")):
    print(f"{os.path.basename(path):32} {os.path.getsize(path) / 1e3:8.1f} kB")

poland_two_layers.gpkg              778.2 kB
portugal_excel_export.csv             2.0 kB
portugal_national_grid.geojson      387.4 kB
portugal_no_prj.zip                 159.3 kB
portugal_shapefile.zip              159.5 kB
portugal_workbook_renamed.csv         6.8 kB

What the reader does with them, unguarded:

for path in sorted(glob.glob("uploads/*")):
    try:
        gdf = gpd.read_file(path)
        geometry = "no geometry" if gdf.geometry.name not in gdf else \
            f"bounds {gdf.total_bounds.round(1).tolist()}"
        print(f"{os.path.basename(path):32} read {len(gdf)} rows, crs {gdf.crs}, {geometry}")
    except Exception as exc:
        print(f"{os.path.basename(path):32} {type(exc).__name__}: {str(exc).splitlines()[0][:70]}")

poland_two_layers.gpkg           read 73 rows, crs EPSG:4326, bounds [14.1, 49.0, 24.1, 54.8]
portugal_excel_export.csv        UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 1: invalid continuati
portugal_national_grid.geojson   read 25 rows, crs EPSG:4326, bounds [-2001550.3, -1042626.1, 162112.5, 276095.8]
portugal_no_prj.zip              read 25 rows, crs None, bounds [-31.3, 30.0, -6.2, 42.2]
portugal_shapefile.zip           read 25 rows, crs EPSG:4326, bounds [-31.3, 30.0, -6.2, 42.2]
portugal_workbook_renamed.csv    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6 in position 1: invalid continuati

Example 1 — the full upload pipeline

import io
import os
import tempfile
import zipfile

import geopandas as gpd
import pandas as pd
import streamlit as st

MAX_UPLOAD_MB = 50
MAX_FEATURES = 100_000


def handle_upload(upload):
    """Returns (gdf, notes) or (None, problems). Never raises at the user."""
    notes, problems = [], []

    size_mb = upload.size / 1e6
    if size_mb > MAX_UPLOAD_MB:
        return None, [f"{size_mb:.0f} MB exceeds the {MAX_UPLOAD_MB} MB limit"]

    payload = upload.getvalue()
    kind = sniff(payload[:64])
    name = upload.name.lower()

    try:
        if kind == "zip" or name.endswith(".zip"):
            path, shp_problems = extract_shapefile(upload)
            notes += shp_problems
            if path is None:
                return None, shp_problems
            gdf = gpd.read_file(path)
        elif name.endswith((".gpkg",)) or kind == "gpkg":
            with tempfile.NamedTemporaryFile(suffix=".gpkg", delete=False) as handle:
                handle.write(payload)
            layers = gpd.list_layers(handle.name)
            if len(layers) > 1:
                notes.append(f"{len(layers)} layers; using {layers.name.iloc[0]!r}")
            gdf = gpd.read_file(handle.name, layer=layers.name.iloc[0])
        elif name.endswith((".geojson", ".json")) or kind == "json":
            gdf = gpd.read_file(io.BytesIO(payload))
        elif name.endswith(".parquet") or kind == "parquet":
            gdf = gpd.read_parquet(io.BytesIO(payload))
        elif name.endswith(".csv"):
            return read_csv_upload(payload)
        else:
            return None, [f"unrecognised file type ({kind})"]
    except UnicodeDecodeError as exc:
        return None, [f"the text encoding is not UTF-8 ({exc}). "
                      f"Re-export as UTF-8, or include a .cpg file."]
    except Exception as exc:                                # noqa: BLE001
        return None, [f"could not read the file: "
                      f"{str(exc).splitlines()[0][:200]}"]

    if len(gdf) > MAX_FEATURES:
        return None, [f"{len(gdf):,} features exceeds the {MAX_FEATURES:,} limit"]

    gdf, crs_note = resolve_crs(gdf)
    if gdf is None:
        return None, [crs_note]
    if crs_note:
        notes.append(crs_note)

    gdf, report = geometry_report(gdf)
    if report["invalid"]:
        notes.append(f"{report['invalid']} invalid geometries were repaired")
    if report["null"] or report["empty"]:
        notes.append(f"{report['null'] + report['empty']} empty or null "
                     f"geometries were dropped")
    if len(gdf) == 0:
        return None, ["no usable geometries after validation"]

    return gdf, notes

Example 2 — a CSV upload with column selection

def read_csv_upload(payload: bytes):
    for encoding in ("utf-8-sig", "utf-8", "cp1252", "latin-1"):
        try:
            frame = pd.read_csv(io.BytesIO(payload), encoding=encoding)
            break
        except UnicodeDecodeError:
            continue
    else:
        return None, ["could not decode the CSV in any common encoding"]

    lon_guess, lat_guess = coordinate_columns(frame)
    columns = list(frame.columns)

    lon = st.selectbox("Longitude / easting column", columns,
                       index=columns.index(lon_guess) if lon_guess else 0)
    lat = st.selectbox("Latitude / northing column", columns,
                       index=columns.index(lat_guess) if lat_guess else 0)
    epsg = st.number_input("EPSG code", value=4326, step=1)

    coordinates = frame[[lon, lat]].apply(pd.to_numeric, errors="coerce")
    bad = int(coordinates.isna().any(axis=1).sum())
    frame = frame[~coordinates.isna().any(axis=1)]
    coordinates = coordinates.dropna()

    gdf = gpd.GeoDataFrame(
        frame,
        geometry=gpd.points_from_xy(coordinates[lon], coordinates[lat]),
        crs=int(epsg)).to_crs(4326)

    notes = [f"{bad} rows had unparseable coordinates and were dropped"] if bad else []
    return gdf, notes

Making the columns and the EPSG code explicit widgets, pre-filled with a guess, is what makes the upload work for the file the guess gets wrong.

Streamlit's UploadedFile has a name, a size and getvalue(), which is all the pipeline touches — so a stand-in with the same three runs it outside an app:

class Upload:
    def __init__(self, path):
        self.name = os.path.basename(path)
        with open(path, "rb") as handle:
            self._data = handle.read()
        self.size = len(self._data)

    def getvalue(self):
        return self._data


for path in sorted(glob.glob("uploads/*")):
    gdf, messages = handle_upload(Upload(path))
    verdict = f"loaded {len(gdf)} features" if gdf is not None else "refused"
    print(f"{os.path.basename(path):32} {verdict}")
    for message in messages:
        print(f"    {message}")

poland_two_layers.gpkg           loaded 73 features
    2 layers; using 'nuts3'
    1 invalid geometries were repaired
portugal_excel_export.csv        loaded 25 features
portugal_national_grid.geojson   refused
    the coordinates [-2001550.0, -1042626.0, 162112.0, 276096.0] are not lon/lat although the file says EPSG:4326. Tell the app which EPSG code this is.
portugal_no_prj.zip              loaded 25 features
    .prj is missing — the CRS will be unknown
    no CRS declared; the coordinates look like lon/lat, assumed EPSG:4326
portugal_shapefile.zip           loaded 25 features
portugal_workbook_renamed.csv    refused
    this is an Excel workbook, not a CSV — save it as CSV (UTF-8) and upload that

And why the size check comes first — what reading a GeoJSON costs in memory, measured in a fresh process:

import subprocess
import sys

districts.to_file("staging/all_regions.geojson")
probe = """
import geopandas as gpd

def rss_mb():
    with open("/proc/self/status") as status:
        line = next(l for l in status if l.startswith("VmRSS"))
    return int(line.split()[1]) / 1024

before = rss_mb()
gdf = gpd.read_file("staging/all_regions.geojson")
print(rss_mb() - before)
"""
grown = float(subprocess.run([sys.executable, "-c", probe], capture_output=True,
                             text=True, check=True).stdout)
size = os.path.getsize("staging/all_regions.geojson") / 1e6
print(f"{size:.1f} MB GeoJSON: the process grew by {grown:.0f} MB, {grown / size:.1f}× the file")

27.0 MB GeoJSON: the process grew by 65 MB, 2.4× the file

Example 3 — reporting the outcome to the user

def show_upload_result(gdf, notes):
    st.success(f"Loaded {len(gdf):,} features · "
               f"{gdf.geom_type.value_counts().to_dict()} · "
               f"{gdf.crs.to_string()}")
    minx, miny, maxx, maxy = gdf.total_bounds
    st.caption(f"Extent: {minx:.3f}, {miny:.3f} → {maxx:.3f}, {maxy:.3f}")

    for note in notes:
        st.warning(note)

    with st.expander("Columns"):
        st.dataframe(
            pd.DataFrame({"column": gdf.columns,
                          "dtype": [str(t) for t in gdf.dtypes],
                          "nulls": [int(gdf[c].isna().sum()) for c in gdf.columns]}),
            width="stretch")

Showing the extent is worth as much as the feature count: a user whose file lands in the wrong place sees it immediately, before the map draws.

Why an upload is untrusted input

Why an upload is untrusted input

Every assumption a reader makes — encoding, CRS, geometry validity, column names, file type — is a claim the file may not honour. Users upload what they have, which is frequently an export from a system nobody in the conversation controls.

Treating the upload as validated input is how an app ends up showing a stack trace to a person who wanted to look at a map. A validation pipeline turns each broken assumption into a sentence they can act on.

Why a missing .prj is the most consequential omission

Without it the file reads and produces coordinates whose meaning is undefined. If they happen to fall in the lon/lat range, an app that assumes EPSG:4326 will draw them somewhere plausible and wrong.

That is why the resolver reports its assumption: "no CRS declared; the coordinates look like lon/lat, assumed EPSG:4326" is a statement the user can contradict. A silent assumption is not.

Why size limits belong in two places

The platform limit refuses the transfer; the application limit refuses the work. Without the first, a 400 MB upload is transferred and buffered before anything can reject it. Without the second, a large GeoJSON is accepted and read — measured here at 2.4 times its file size in memory, before any copy, reprojection or rendering — and can take the session down with it.

Setting both means the user gets a message rather than a disconnection, and the server is not holding a file it will not use.

Why the guess must be overridable

Column-name detection works until it meets a file with x and y columns holding eastings and northings, or lat and lon reversed, or coordinates in a national grid.

Offering the guess as a pre-filled widget keeps the common case one click and makes the uncommon case possible. Automatic detection with no override turns a five-second correction into an unusable app.

Upload limits, sidecars, layers and encodings

• Set --server.maxUploadSize as well as an application check.
• Sniff the file type by content; an extension is a claim.
• A zipped shapefile may be missing .prj or .dbf — read but degraded.
• A GeoPackage can hold several layers; ask, or say which one you used.
• CSV encodings: try utf-8-sig first, then cp1252; latin-1 never fails and often lies.
• Report the extent, not only the feature count.
• Repair invalid geometry and say so — a silent repair surprises people later.
• Uploaded data is per session, so cap the feature count as well as the file size.

Internal links

• How to build a Streamlit app with an interactive map — where the upload sits
• How to open any spatial file in Python — the reading layer
• How to validate a GeoDataFrame schema — checking what arrived
• Fixing shapefile encoding errors in Python — the .cpg problem
• Fixing invalid geometries in GeoPandas — the repair step
• How to clean CSV coordinates into a GeoDataFrame — the CSV path
• Fixing an app whose memory grows with every user — why uploads need a cap
• Fixing blank or grey tiles in an embedded map — when the upload loads and nothing draws

FAQ

How do I accept a shapefile upload?

Ask for a zip, extract it to a temporary directory, and check for .shx, .dbf and .prj before reading. Report which are missing and what that costs.

What should I do when the upload has no CRS?

Say so. If the coordinates fall in the lon/lat range you may assume EPSG:4326 — but report the assumption, and let the user supply the correct code.

How large an upload should I allow?

Set both the platform limit (--server.maxUploadSize) and an application check. Measured, a 27 MB GeoJSON took 65 MB of memory once read, before any copy, reprojection or rendering — and it is held per session.

Why does my CSV upload fail with a decode error?

The file is not UTF-8. Try utf-8-sig, then cp1252; avoid falling back to latin-1, which never fails and silently produces mojibake.

Should I repair invalid geometry automatically?

Repair it and report it. "1 invalid geometry was repaired" is useful; a silent repair produces a surprise when the areas do not match.

How do I find the coordinate columns in a CSV?

Guess from the column names and present the guess in two selectboxes the user can change. Detection without an override fails on the first file with eastings named x.