Fixing an Uploaded File That Breaks a Map App
Problem statement
The app accepts a file upload, and users send everything: a zipped shapefile missing its .prj, a CSV with coordinates in three different formats, a 400 MB GeoPackage, a GeoJSON in a national grid, and a spreadsheet renamed to .csv.
Each produces a different failure and most of them produce the same symptom — a stack trace in the browser, or a blank map with no explanation:
DataSourceError: not recognized as a supported file format
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6
ValueError: cannot convert float NaN to integer
MemoryError
An upload is untrusted input. The fix is a validation pipeline that checks the file before anything tries to render it, and reports what is wrong in terms the user can act on.
Quick answer
Validate in stages, and fail with a message rather than a trace:
import streamlit as st
MAX_UPLOAD_MB = 50
SUPPORTED = {".geojson", ".json", ".gpkg", ".zip", ".csv", ".parquet"}
upload = st.file_uploader("Upload a layer",
type=[e.lstrip(".") for e in SUPPORTED])
if upload is not None:
problems = validate_upload(upload)
if problems:
st.error("This file cannot be used:")
for problem in problems:
st.write(f"- {problem}")
st.stop()
gdf = read_upload(upload)
st.success(f"Loaded {len(gdf):,} features in {gdf.crs.to_string()}")Also set the platform limit, because the default is generous:
streamlit run app.py --server.maxUploadSize=50
Step-by-step solution
1. Bound the size before reading anything
Streamlit's default upload limit is 200 MB, and a 200 MB GeoJSON expands to several times that in memory — measured, 4,596 polygons alone are 54 MB as GeoJSON.
size_mb = upload.size / 1e6
if size_mb > MAX_UPLOAD_MB:
st.error(f"{size_mb:.0f} MB exceeds the {MAX_UPLOAD_MB} MB limit. "
f"Upload a filtered extract, or use the bulk import.")
st.stop()Set --server.maxUploadSize as well, so the transfer is refused rather than completed and then rejected.
2. Check the type by content, not by extension
An extension is a claim. Read the first bytes:
MAGIC = {b"PK\x03\x04": "zip", b"SQLite format 3\x00": "gpkg",
b"PAR1": "parquet"}
def sniff(head: bytes) -> str:
for signature, kind in MAGIC.items():
if head.startswith(signature):
return kind
text = head.lstrip()[:1]
if text in (b"{", b"["):
return "json"
return "text"A .geojson that is actually a zip, or a .csv that is an Excel workbook, is common enough to be worth two lines.
3. Handle a zipped shapefile properly
A shapefile is several files, so it arrives zipped — and frequently incomplete:
import zipfile, io, tempfile, os
def extract_shapefile(upload) -> tuple[str | None, list[str]]:
problems = []
with zipfile.ZipFile(io.BytesIO(upload.getvalue())) as archive:
names = archive.namelist()
shp = [n for n in names if n.lower().endswith(".shp")]
if not shp:
return None, ["the zip contains no .shp file"]
if len(shp) > 1:
problems.append(f"{len(shp)} shapefiles in the zip; using the first")
stem = shp[0][:-4]
for suffix, consequence in ((".shx", "required"),
(".dbf", "attributes will be missing"),
(".prj", "the CRS will be unknown")):
if stem + suffix not in names:
problems.append(f"{suffix} is missing — {consequence}")
directory = tempfile.mkdtemp()
archive.extractall(directory)
return os.path.join(directory, shp[0]), problemsThe .prj warning is the important one: without it the file reads and its coordinates mean nothing definite.
4. Establish the CRS, and refuse to guess silently
def resolve_crs(gdf, declared_epsg=None):
if gdf.crs is not None:
return gdf.to_crs(4326), None
if declared_epsg:
return gdf.set_crs(declared_epsg, allow_override=True).to_crs(4326), \
f"assumed EPSG:{declared_epsg} as declared"
minx, miny, maxx, maxy = gdf.total_bounds
if -180 <= minx <= 180 and -90 <= miny <= 90:
return gdf.set_crs(4326).to_crs(4326), \
"no CRS declared; the coordinates look like lon/lat, assumed EPSG:4326"
return None, (f"no CRS, and the bounds {gdf.total_bounds} are not lon/lat. "
f"Tell the app which EPSG code this is.")Guessing from the bounds is defensible when it is reported; guessing silently is how a layer ends up plotted in the Gulf of Guinea.
5. Validate the geometry
def geometry_report(gdf):
report = {
"features": len(gdf),
"null": int(gdf.geometry.isna().sum()),
"empty": int(gdf.geometry.is_empty.sum()),
"invalid": int((~gdf.geometry.is_valid).sum()),
"types": gdf.geom_type.value_counts().to_dict(),
}
if report["invalid"]:
gdf = gdf.assign(geometry=gdf.geometry.make_valid())
return gdf[~gdf.geometry.isna() & ~gdf.geometry.is_empty], reportReport the counts to the user. "3 of 1,240 features had invalid geometry and were repaired" is information; a silent repair is a surprise later.
6. Handle a CSV's coordinate columns explicitly
LON_NAMES = {"lon", "long", "longitude", "x", "easting"}
LAT_NAMES = {"lat", "latitude", "y", "northing"}
def coordinate_columns(frame):
lowered = {c.lower(): c for c in frame.columns}
lon = next((lowered[n] for n in LON_NAMES if n in lowered), None)
lat = next((lowered[n] for n in LAT_NAMES if n in lowered), None)
return lon, latOffer the guess and let the user correct it with two selectboxes. Automatic detection that cannot be overridden fails on the first file with x and y columns that are eastings and northings.
Code examples
Example 1 — the full upload pipeline
import io
import os
import tempfile
import zipfile
import geopandas as gpd
import pandas as pd
import streamlit as st
MAX_UPLOAD_MB = 50
MAX_FEATURES = 100_000
def handle_upload(upload):
"""Returns (gdf, notes) or (None, problems). Never raises at the user."""
notes, problems = [], []
size_mb = upload.size / 1e6
if size_mb > MAX_UPLOAD_MB:
return None, [f"{size_mb:.0f} MB exceeds the {MAX_UPLOAD_MB} MB limit"]
payload = upload.getvalue()
kind = sniff(payload[:64])
name = upload.name.lower()
try:
if kind == "zip" or name.endswith(".zip"):
path, shp_problems = extract_shapefile(upload)
notes += shp_problems
if path is None:
return None, shp_problems
gdf = gpd.read_file(path)
elif name.endswith((".gpkg",)) or kind == "gpkg":
with tempfile.NamedTemporaryFile(suffix=".gpkg", delete=False) as handle:
handle.write(payload)
layers = gpd.list_layers(handle.name)
if len(layers) > 1:
notes.append(f"{len(layers)} layers; using {layers.name.iloc[0]!r}")
gdf = gpd.read_file(handle.name, layer=layers.name.iloc[0])
elif name.endswith((".geojson", ".json")) or kind == "json":
gdf = gpd.read_file(io.BytesIO(payload))
elif name.endswith(".parquet") or kind == "parquet":
gdf = gpd.read_parquet(io.BytesIO(payload))
elif name.endswith(".csv"):
return read_csv_upload(payload)
else:
return None, [f"unrecognised file type ({kind})"]
except UnicodeDecodeError as exc:
return None, [f"the text encoding is not UTF-8 ({exc}). "
f"Re-export as UTF-8, or include a .cpg file."]
except Exception as exc: # noqa: BLE001
return None, [f"could not read the file: "
f"{str(exc).splitlines()[0][:200]}"]
if len(gdf) > MAX_FEATURES:
return None, [f"{len(gdf):,} features exceeds the {MAX_FEATURES:,} limit"]
gdf, crs_note = resolve_crs(gdf)
if gdf is None:
return None, [crs_note]
if crs_note:
notes.append(crs_note)
gdf, report = geometry_report(gdf)
if report["invalid"]:
notes.append(f"{report['invalid']} invalid geometries were repaired")
if report["null"] or report["empty"]:
notes.append(f"{report['null'] + report['empty']} empty or null "
f"geometries were dropped")
if len(gdf) == 0:
return None, ["no usable geometries after validation"]
return gdf, notesExample 2 — a CSV upload with column selection
def read_csv_upload(payload: bytes):
for encoding in ("utf-8-sig", "utf-8", "cp1252", "latin-1"):
try:
frame = pd.read_csv(io.BytesIO(payload), encoding=encoding)
break
except UnicodeDecodeError:
continue
else:
return None, ["could not decode the CSV in any common encoding"]
lon_guess, lat_guess = coordinate_columns(frame)
columns = list(frame.columns)
lon = st.selectbox("Longitude / easting column", columns,
index=columns.index(lon_guess) if lon_guess else 0)
lat = st.selectbox("Latitude / northing column", columns,
index=columns.index(lat_guess) if lat_guess else 0)
epsg = st.number_input("EPSG code", value=4326, step=1)
coordinates = frame[[lon, lat]].apply(pd.to_numeric, errors="coerce")
bad = int(coordinates.isna().any(axis=1).sum())
frame = frame[~coordinates.isna().any(axis=1)]
coordinates = coordinates.dropna()
gdf = gpd.GeoDataFrame(
frame,
geometry=gpd.points_from_xy(coordinates[lon], coordinates[lat]),
crs=int(epsg)).to_crs(4326)
notes = [f"{bad} rows had unparseable coordinates and were dropped"] if bad else []
return gdf, notesMaking the columns and the EPSG code explicit widgets, pre-filled with a guess, is what makes the upload work for the file the guess gets wrong.
Example 3 — reporting the outcome to the user
def show_upload_result(gdf, notes):
st.success(f"Loaded {len(gdf):,} features · "
f"{gdf.geom_type.value_counts().to_dict()} · "
f"{gdf.crs.to_string()}")
minx, miny, maxx, maxy = gdf.total_bounds
st.caption(f"Extent: {minx:.3f}, {miny:.3f} → {maxx:.3f}, {maxy:.3f}")
for note in notes:
st.warning(note)
with st.expander("Columns"):
st.dataframe(
pd.DataFrame({"column": gdf.columns,
"dtype": [str(t) for t in gdf.dtypes],
"nulls": [int(gdf[c].isna().sum()) for c in gdf.columns]}),
use_container_width=True)Showing the extent is worth as much as the feature count: a user whose file lands in the wrong place sees it immediately, before the map draws.
Explanation
Why an upload is untrusted input
Every assumption a reader makes — encoding, CRS, geometry validity, column names, file type — is a claim the file may not honour. Users upload what they have, which is frequently an export from a system nobody in the conversation controls.
Treating the upload as validated input is how an app ends up showing a stack trace to a person who wanted to look at a map. A validation pipeline turns each broken assumption into a sentence they can act on.
Why a missing .prj is the most consequential omission
Without it the file reads and produces coordinates whose meaning is undefined. If they happen to fall in the lon/lat range, an app that assumes EPSG:4326 will draw them somewhere plausible and wrong.
That is why the resolver reports its assumption: "no CRS declared; the coordinates look like lon/lat, assumed EPSG:4326" is a statement the user can contradict. A silent assumption is not.
Why size limits belong in two places
The platform limit refuses the transfer; the application limit refuses the work. Without the first, a 400 MB upload is transferred and buffered before anything can reject it. Without the second, a 40 MB GeoJSON that expands to several hundred megabytes in memory is accepted and kills the session.
Setting both means the user gets a message rather than a disconnection, and the server is not holding a file it will not use.
Why the guess must be overridable
Column-name detection works until it meets a file with x and y columns holding eastings and northings, or lat and lon reversed, or coordinates in a national grid.
Offering the guess as a pre-filled widget keeps the common case one click and makes the uncommon case possible. Automatic detection with no override turns a five-second correction into an unusable app.
Edge cases or notes
- Set
--server.maxUploadSizeas well as an application check. - Sniff the file type by content; an extension is a claim.
- A zipped shapefile may be missing
.prjor.dbf— read but degraded. - A GeoPackage can hold several layers; ask, or say which one you used.
- CSV encodings: try
utf-8-sigfirst, thencp1252;latin-1never fails and often lies. - Report the extent, not only the feature count.
- Repair invalid geometry and say so — a silent repair surprises people later.
- Uploaded data is per session, so cap the feature count as well as the file size.
Internal links
- How to build a Streamlit app with an interactive map — where the upload sits
- How to open any spatial file in Python — the reading layer
- How to validate a GeoDataFrame schema — checking what arrived
- Fixing shapefile encoding errors in Python — the
.cpgproblem - Fixing invalid geometries in GeoPandas — the repair step
- How to clean CSV coordinates into a GeoDataFrame — the CSV path
- Fixing an app whose memory grows with every user — why uploads need a cap
- Fixing blank or grey tiles in an embedded map — when the upload loads and nothing draws
FAQ
How do I accept a shapefile upload?
Ask for a zip, extract it to a temporary directory, and check for .shx, .dbf and .prj before reading. Report which are missing and what that costs.
What should I do when the upload has no CRS?
Say so. If the coordinates fall in the lon/lat range you may assume EPSG:4326 — but report the assumption, and let the user supply the correct code.
How large an upload should I allow?
Set both the platform limit (--server.maxUploadSize) and an application check. A 40 MB GeoJSON expands to several times that in memory, and it is held per session.
Why does my CSV upload fail with a decode error?
The file is not UTF-8. Try utf-8-sig, then cp1252; avoid falling back to latin-1, which never fails and silently produces mojibake.
Should I repair invalid geometry automatically?
Repair it and report it. "3 of 1,240 features were repaired" is useful; a silent repair produces a surprise when the areas do not match.
How do I find the coordinate columns in a CSV?
Guess from the column names and present the guess in two selectboxes the user can change. Detection without an override fails on the first file with eastings named x.