Fixing Coordinates That Lose Precision When Saved to a File

Problem statement

Points that were on the right building yesterday are now on the neighbour's roof, or a few metres into the street, after a round trip through a file. Nothing moved; precision was thrown away. Two silent conversions do it: rounding to a fixed number of decimal places when writing text, and storing coordinates as 32-bit floats, which keep only about seven significant digits, so a longitude of −157.858333 loses its last digits. Each is invisible in a table and visible on a map at street scale.

This page measures both losses on the 80,000 coordinates in OurAirports: the error introduced by rounding to two to six decimals, the error of float32 storage at different longitudes, a CSV and a Parquet round trip done badly and done right, and how to spot data that has already been truncated.

Quick answer

Write coordinates as 64-bit floats with enough decimals, and check the round trip:

df = A[["latitude_deg", "longitude_deg"]]
df.astype("float32").to_csv("data/coords_f32.csv", index=False)
df.to_csv("data/coords_f64.csv", index=False, float_format="%.6f")
for name in ("coords_f32.csv", "coords_f64.csv"):
    back = pd.read_csv(f"data/{name}")
    err = geod.inv(df.longitude_deg.values, df.latitude_deg.values, back.longitude_deg.values, back.latitude_deg.values)[2]
    print(f"{name}: max error {err.max():.3f} m, median {np.median(err):.3f} m")

coords_f32.csv: max error 1.623 m, median 0.141 m
coords_f64.csv: max error 0.075 m, median 0.000 m

Written as float32, the CSV round trip moves points by a median 0.14 m and up to 1.62 m. Written as float64 with six decimals, the median displacement is zero and the largest 7.5 cm, the rounding of the sixth decimal.

Fixed decimals, float32 at large values, truncated data

1. Rounding to fixed decimals

Round every coordinate to a given number of decimals and measure how far it moves:

for dec in (2, 3, 4, 5, 6):
    lat_r, lon_r = df.latitude_deg.round(dec), df.longitude_deg.round(dec)
    err = geod.inv(df.longitude_deg.values, df.latitude_deg.values, lon_r.values, lat_r.values)[2]
    print(f"{dec} decimals: median {np.median(err):9.3f} m, 99th percentile {np.percentile(err, 99):9.3f} m, max {err.max():9.3f} m")

2 decimals: median   394.995 m, 99th percentile   685.587 m, max   782.998 m
3 decimals: median    38.598 m, 99th percentile    68.502 m, max    78.449 m
4 decimals: median     3.082 m, 99th percentile     6.759 m, max     7.843 m
5 decimals: median     0.262 m, 99th percentile     0.688 m, max     0.785 m
6 decimals: median     0.000 m, 99th percentile     0.061 m, max     0.075 m

Rounding to four decimals moves points by a median 3 m and at most 7.8 m; three decimals by up to 78 m, enough to put a point on the wrong side of a road; two decimals by up to 783 m. Five decimals keep every point within 0.8 m, and six within 7.5 cm.

2. Float32 loses more at larger values

A float32 has a 24-bit mantissa, so its spacing grows with the magnitude of the number. Compare the spacing for latitudes and longitudes:

for v in (1.0, 45.0, 90.0, 157.86, 179.99):
    step = np.spacing(np.float32(v))
    print(f"value {v:>7}: float32 spacing {step:.2e}° = {step * 111_320:.3f} m (at the equator); float64 {np.spacing(v) * 111_320:.2e} m")
f32 = df.astype("float32").astype("float64")
err32 = geod.inv(df.longitude_deg.values, df.latitude_deg.values, f32.longitude_deg.values, f32.latitude_deg.values)[2]
print(f"all places as float32: median error {np.median(err32):.3f} m, max {err32.max():.3f} m")

value     1.0: float32 spacing 1.19e-07° = 0.013 m (at the equator); float64 2.47e-11 m
value    45.0: float32 spacing 3.81e-06° = 0.425 m (at the equator); float64 7.91e-10 m
value    90.0: float32 spacing 7.63e-06° = 0.849 m (at the equator); float64 1.58e-09 m
value  157.86: float32 spacing 1.53e-05° = 1.699 m (at the equator); float64 3.16e-09 m
value  179.99: float32 spacing 1.53e-05° = 1.699 m (at the equator); float64 3.16e-09 m
all places as float32: median error 0.108 m, max 0.843 m

The spacing of float32 values grows with the number: about 1 cm near 1°, 0.85 m near 90° and 1.7 m near 158° or 180°, measured at the equator. Stored as float32, the places move by a median 0.11 m and up to 0.84 m, half the spacing at large longitudes. The CSV round trip above did worse, 1.62 m, because writing float32 values as text added its own rounding.

3. Spot data that was already truncated

Count the decimal places in the text of a coordinate column. A source with every value at four decimals or fewer has been rounded, whatever its documentation says:

raw = pd.read_csv("data/airports.csv", dtype=str, keep_default_na=False)
decs = raw.longitude_deg.str.split(".").str[1].str.len().fillna(0).astype(int)
print("decimal places in OurAirports longitudes:", decs.value_counts().sort_index().to_dict())
print(f"places with 4 or fewer decimals: {(decs <= 4).mean():.1%}")

decimal places in OurAirports longitudes: {0: 48, 1: 101, 2: 424, 3: 1352, 4: 5795, 5: 17512, 6: 46167, 7: 540, 8: 327, 9: 926, 10: 835, 11: 230, 12: 408, 13: 1663, 14: 8873, 15: 849, 16: 93, 17: 12, 18: 1}
places with 4 or fewer decimals: 9.0%

OurAirports itself mixes precisions: most longitudes have five or six decimals, 9.0% have four or fewer, and about 11,400 have 13 to 15, the trace of values computed or converted somewhere upstream rather than measured that precisely.

Parquet keeps the type you give it

The data used on this page

Every open place in OurAirports, a public-domain database, with coordinates as published in its CSV. Displacements are geodesic distances from pyproj.Geod on WGS84.

from pathlib import Path

import numpy as np
import pandas as pd
import requests
from pyproj import Geod

Path("data").mkdir(exist_ok=True)
p = Path("data/airports.csv")
if not p.exists():
    r = requests.get("https://davidmegginson.github.io/ourairports-data/airports.csv", timeout=300)
    r.raise_for_status()
    p.write_bytes(r.content)
A = pd.read_csv(p, keep_default_na=False, na_values=[""])
A = A[A.type != "closed"].reset_index(drop=True)
geod = Geod(ellps="WGS84")
print(f"{len(A):,} places; dtypes {A.latitude_deg.dtype}, {A.longitude_deg.dtype}")

72,603 places; dtypes float64, float64

OurAirports' 72,603 open places load as float64.

Example 1: Parquet keeps the type you give it

Binary formats store exactly the dtype of the column, which is good with float64 and bad if a column was downcast to save memory:

df.to_parquet("data/coords_f64.parquet"); df.astype("float32").to_parquet("data/coords_f32.parquet")
for name in ("coords_f64.parquet", "coords_f32.parquet"):
    back = pd.read_parquet(f"data/{name}")
    err = geod.inv(df.longitude_deg.values, df.latitude_deg.values, back.longitude_deg.values.astype(float), back.latitude_deg.values.astype(float))[2]
    print(f"{name}: dtype {back.longitude_deg.dtype}, size {Path('data/' + name).stat().st_size / 1e3:,.0f} kB, max error {err.max():.4f} m")

coords_f64.parquet: dtype float64, size 1,387 kB, max error 0.0000 m
coords_f32.parquet: dtype float32, size 862 kB, max error 0.8426 m

Parquet stores exactly the dtype it is given: the float64 file round-trips without any error at 1,387 kB; the float32 file is 862 kB, 38% smaller, and moves points by up to 0.84 m.

What float64 and float32 can carry

A float64 carries about 15 to 16 significant decimal digits, far more than any coordinate needs: its spacing near 180 is about 3 × 10⁻¹⁴ degrees, a few nanometres. A float32 carries about 7: near 180 its spacing is about 1.5 × 10⁻⁵ degrees, close to 2 m at the equator, and a value is stored to within half of that. Latitudes, never above 90, lose less than longitudes, which reach 180. That is why the same downcast that is harmless for a temperature moves a point by a metre or more, and why the error is larger in the Pacific and the Americas, at large longitudes, than near Greenwich.

Rounding text is the other path. Each decimal place of a degree is a factor of ten: four decimals keep about 11 m, five about 1 m, six about 10 cm. Writing coordinates with float_format="%.3f" to make a file tidier, or letting a spreadsheet save the numbers as displayed, cuts them to the displayed precision for good. Six decimals are enough for almost any application; more than seven is noise from the source's computations.

GeoJSON writers and truncation

• GeoJSON writers: many writers let you choose the coordinate precision; six decimals is a common default for web maps and is about 10 cm.
• Memory: downcasting other columns to float32 is fine; keep coordinates in float64.
• Projected coordinates: eastings and northings in metres reach 10⁶ to 10⁷, so float32 loses 0.06 to 1 m there too.
• Truncation vs rounding: some tools truncate instead of rounding, which biases every point in one direction.

Internal links

• How many decimal places do coordinates need? — choosing the precision
• How to convert degrees, minutes and seconds to decimal degrees in Python — precision of DMS sources
• Latitude and longitude explained — what a degree is on the ground

FAQ

Why did my coordinates move after saving to CSV?

They were rounded to fewer decimals when written, or stored as 32-bit floats, which keep only about seven significant digits.

Is float32 enough for latitude and longitude?

No: near 180° of longitude it is only precise to about 2 m; use float64.

How many decimals should I write?

Six decimal places, about 10 cm, is enough for almost every use.

How can I tell if coordinates were truncated?

Count the decimal places in the raw text; if every value has the same small number, the data was rounded somewhere upstream.