Data cleaning

Cleaning is where an analysis acquires its assumptions, which is why it deserves to be written down rather than done interactively. Every step is a rule applied to a set of rows, and the rule is the part that matters: "removed 312" tells a reader how many, and only the rule tells them why.

The work itself is unglamorous and specific: coordinate columns stored as text, decimal commas, swapped latitude and longitude, duplicate geometries that differ in the eighth decimal place, categories spelled four ways, missing values encoded as −9999 and as the empty string in the same file. Almost none of it raises an error; it produces a dataset that looks finished.

The pages below cover what to check before trusting a file, the operations that fix each defect, and a repeatable report that makes the whole thing reproducible six months later.

Learning paths run through this topic too — see the end of this page.