Data cleaning
Turning a messy download into an analysis-ready dataset you can trust.
Everything on Spatial Workflow about data cleaning, in the order that usually works: understand the idea, then run the task, then fix it when it goes wrong. 70 guides in total.
13 learning paths run through this topic — follow one if you would rather work through a route than pick a single guide.
18
Concepts
32
How-To guides
20
Fixes
🧭 Start with the concepts
The ideas the how-to guides below assume you already have.
- Spatial Data Quality: The Six Dimensions That Matter spatial data quality dimensions
- Null, Empty, Missing and Invalid: Four Kinds of Broken Geometry empty geometry geopandas
- Topological Coverage: What "Clean" Means for Adjacent Polygons polygon coverage gaps overlaps
- Repair, Reject or Flag? Choosing What Cleaning Should Do gis data cleaning decisions
- Choropleth Classification Explained: Quantiles, Equal Interval and Natural Breaks choropleth classification schemes
- Test Fixtures for GIS Code: Building Spatial Test Data You Can Trust gis test fixtures python
- Spatial Join Cardinality Explained: One-to-One, One-to-Many and What to Do spatial join one to many
- Attribute Join or Spatial Join? Choosing How to Combine Two Layers attribute join vs spatial join
- The Modifiable Areal Unit Problem Explained modifiable areal unit problem
- Spatial Autocorrelation Explained spatial autocorrelation explained
- The Address Data Model: Why an Address Is Not a String address data model
- Rooftop, Interpolated or Centroid: Geocoding Precision Levels geocoding precision levels
- Address Matching Explained: Why Exact String Equality Fails address matching explained
- Census Identifiers Explained: GEOIDs, Codes and the Leading Zero Problem census geoid explained
- Boundary Changes Over Time: Why Two Census Years Do Not Line Up census boundary changes over time
- Geoprivacy explained: why coordinates are personal data geospatial privacy
- Spatial metadata explained: what a dataset must tell you spatial metadata
- Yield monitor data explained: what the numbers really are yield monitor data
⚙️ Then do the task
Step-by-step walkthroughs with code you can copy and adapt.
- How to Remove Null and Empty Geometries in GeoPandas remove null geometries geopandas
- The Python GIS Data Cleaning Checklist: From Raw Download to Analysis-Ready gis data cleaning python
- How to Find and Remove Duplicate Geometries in GeoPandas remove duplicate geometries geopandas
- How to Clean Messy CSV Coordinates into a Reliable GeoDataFrame clean csv coordinates
- How to Standardise and Repair CRS Across a Folder of Files in Python standardise crs across files python
- How to Clean and Normalise Attribute Columns in a GeoDataFrame clean attribute columns geodataframe
- How to Handle Missing and Null Values in Spatial Datasets in Python handle missing values spatial data python
- How to Snap and Align Geometries to Fix Slivers and Gaps in Python snap geometries fix slivers python
- How to Validate a GeoDataFrame Against a Schema Before Analysis validate geodataframe schema python
- How to Build a Repeatable Data-Cleaning Report in GeoPandas data cleaning report geopandas
- How to Split Multipart Geometries into Single Parts in GeoPandas explode multipart geometries geopandas
- How to Fuzzy-Match Place Names When Joining Spatial Data in Python fuzzy match place names python
- How to Find and Remove Spatial Outliers in a Point Dataset remove spatial outliers python
- How to Clean and Standardise Date Columns in Spatial Data clean date columns spatial data python
- How to Explore a Spatial Dataset You Have Never Seen Before explore spatial dataset python
- How to Find and Fix Gaps and Overlaps in a Polygon Coverage fix gaps overlaps polygons python
- How to Merge Near-Duplicate Features in a Spatial Dataset merge near duplicate features python
- How to Clean a Line Network: Dangles, Overshoots and Disconnects clean road network topology python
- How to Standardise Attribute Values Against a Controlled Vocabulary standardise attribute values python
- How to Set, Assign and Convert CRS in GeoPandas (set_crs vs to_crs) set_crs vs to_crs geopandas
- How to Reproject Between Datums Correctly (and Why the Grid File Matters) datum transformation python pyproj
- How to Geocode Addresses in Python geocode addresses python
- How to Clean a GPS Track: Outliers, Duplicates and Jumps clean gps track python
- How to Resample and Interpolate a Track to a Fixed Interval resample gps track python
- How to Parse and Normalise Addresses in Python parse addresses python
- How to Validate Geocoding Results Before You Trust Them validate geocoding results
- How to Match Addresses Against a Reference File in Python match addresses reference file
- How to Deduplicate an Address List in Python deduplicate addresses python
- How to Join a Census Table to Its Boundaries Without Losing Rows join census data to shapefile python
- How to Normalise Census Counts into Rates and Densities normalize census data per capita python
- How to measure shoreline change between two dates shoreline change analysis python
- How to clean yield monitor data in Python clean yield monitor data python
🔧 When it breaks
The errors this topic produces, with the cause explained rather than just the command.
- How to Fix Invalid Geometries in Python (GeoPandas) invalid geometries geopandas
- Why Are My Shapefile Column Names Truncated? How to Fix It shapefile column names truncated
- buffer(0) Does Not Fix My Invalid Geometry: What to Do Instead buffer 0 not fixing geometry
- Garbled Attribute Text and UnicodeDecodeError from a Shapefile: How to Fix Encoding shapefile encoding error python
- GeoPandas to_file() Fails on a Column Type: How to Fix It geopandas to_file field type error
- Rasterio Returns the Wrong Values: NoData, Scaling and dtype Fixes rasterio nodata wrong values
- My Cleaned Layer Has More Rows Than the Original: How to Fix It geopandas more rows after cleaning
- My GeoPandas Choropleth Colours Look Wrong: How to Fix It geopandas choropleth colours wrong
- Geocoding Returns Wrong or Missing Coordinates geocoding wrong coordinates
- My Time Column Sorts or Joins Wrongly spatial time column timezone errors
- My Track Has Impossible Speeds or Teleports track impossible speeds
- Fixing Geocodes That All Land in the Country Centroid geocodes country centroid
- Fixing a Geocoder That Matches the Wrong Town geocoder wrong town
- Fixing NetCDF Values That Look Wrong: scale_factor, _FillValue and Units netcdf scale_factor fill value wrong
- Fixing cftime and Out-of-Bounds Datetime Errors in xarray cftime datetime error xarray
- Fixing Census API Values Like −666666666 and Other Sentinels census api -666666666
- Fixing Census Totals That Do Not Add Up After Aggregation census totals don't match
- Anonymised points still identify individuals anonymised points still identify
- Tide times are off by hours after a join tide times wrong timezone
- A yield map has impossible values and stripes yield map stripes impossible values
Learning paths through this topic
A curated route rather than a list — the guides in the order that builds on itself.
-
Finding & Fetching Data
Get the data before you analyse it — from OSM, web services and satellite catalogues.
Open path → -
Spatial Statistics
Measure the pattern instead of eyeballing the map.
Open path → -
Raster Analysis
Read, reproject, clip and summarise raster data in Python.
Open path → -
Making Maps
Turn a GeoDataFrame into a map someone else can read.
Open path → -
GIS Data Cleaning
Turn messy spatial data into analysis-ready datasets.
Open path → -
Movement & Time
Tracks are not points — the order and the clock are the data.
Open path → -
Geocoding & Addresses
Address text is data too — and it is the messiest data in the file.
Open path → -
Gridded & Climate Data
A raster with a time axis, a calendar and forty years in one file.
Open path → -
Census & Demographics
The most-used open data there is, and the easiest to map wrongly.
Open path → -
Geoprivacy & Anonymisation
A coordinate is an identifier. Release it as if it were one.
Open path → -
Metadata, Licensing & Provenance
A layer nobody can describe, cite or re-run is a layer nobody should use.
Open path → -
Coastal & Marine
The shoreline is a choice of datum, not a line on a map.
Open path → -
Agriculture & Crops
One field, one season, and a dozen ways to measure it wrong.
Open path →
Related topics
Subjects that keep coming up alongside this one, most overlap first.