⚙️ How-To Movement & time Data cleaning Movement & Time · step 6 of 14 Updated

How to Clean a GPS Track: Outliers, Duplicates and Jumps

Problem statement

Cleaning a GPS track means deciding which of its improbable values are errors and which are movement — and every threshold you set changes the answer.

Measured on 13 real traces with a median speed of 4.7 km/h:

filter                    points removed    track length    change
none                                   0       37.86 km
speed <= 200 km/h                     13       37.69 km     -0.44%
speed <= 100 km/h                     42       37.27 km     -1.57%
speed <=  50 km/h                    121       36.19 km     -4.42%
speed <=  29 km/h                    333       35.27 km     -6.84%

The generous filter removes thirteen points and almost no distance. The aggressive one removes 333 points and nearly 7% of the journey — and much of that is genuine vehicle travel, because this dataset mixes walking and driving.

Cleaning too hard is not the safe option.

Quick answer

Clean in a fixed order, flagging rather than deleting:

def clean_track(df, id_col="track_id", max_speed_ms=56.0,
                max_accel_ms2=10.0, gap_s=300):
    df = df.drop_duplicates(subset=[id_col, "time", "x", "y"])
    df = df.sort_values([id_col, "time"]).reset_index(drop=True)

    g = df.groupby(id_col)
    df["dt"] = g["time"].diff().dt.total_seconds()
    df["dist"] = np.hypot(g["x"].diff(), g["y"].diff())
    df["speed"] = df["dist"] / df["dt"].replace(0, np.nan)
    df["accel"] = g["speed"].diff() / df["dt"].replace(0, np.nan)

    df["flag_dup_time"] = df["dt"] == 0
    df["flag_speed"] = df["speed"] > max_speed_ms
    df["flag_accel"] = df["accel"].abs() > max_accel_ms2
    df["leg"] = (df["dt"] > gap_s).groupby(df[id_col]).cumsum()
    return df

Order matters: duplicates first (they create zero intervals), then derived columns, then flags, then leg splitting.

GPS cleaning as an ordered pipeline: duplicates, sort, derive, flag spikes, split at gaps, then collapse stops.
Each step depends on the previous one. Deriving speed before removing duplicates gives infinities.

Step-by-step solution

1. Remove exact duplicates first

A real 60,000-point download contained 1,388 exact (lat, lon, time) duplicates — 2.3%. Each one creates a zero interval and therefore an infinite derived speed.

df = df.drop_duplicates(subset=["track_id", "time", "x", "y"])

Duplicates on (id, time) with different positions are a separate problem: two fixes claiming the same instant. Keep the one with better quality if the file records it, or the first.

2. Fix the identity column before filtering anything

If consecutive rows come from different journeys, every derived speed is meaningless and a speed filter will delete real data at the boundaries. See How to load GPS tracks into Python as trajectories.

3. Flag on speed and acceleration

A single displaced point produces a high speed into it and a high speed out of it. Filtering on speed alone can remove the good neighbour instead of the bad point.

Acceleration disambiguates: a genuine acceleration to 30 m/s takes seconds, so an implied acceleration above about 10 m/s² is an error rather than a manoeuvre.

4. Choose the threshold from the mode, generously

The table at the top is the argument for generosity. Removing the impossible costs 0.44% of the distance; removing the merely surprising costs 6.84% and deletes real driving.

If the dataset mixes modes, filter only on physical impossibility and separate modes as its own step.

5. Iterate, because one spike hides another

Removing a spike changes its neighbours' speeds, which can reveal a second spike. Two or three passes usually converge; if it does not converge in five, the threshold is too tight.

6. Split at gaps, and collapse stops before measuring

A gap is not a straight line, and a stationary period is not travel. Both inflate distance if left alone. See Stops, trips and segmentation explained.

A single displaced point producing a high speed in and out, against a sustained offset where every speed is plausible.
A spike is visible in the acceleration. A sustained offset is not visible at all without external information.

Code examples

Example 1 — iterative spike removal with a report 

import numpy as np


def remove_spikes(df, id_col="track_id", max_speed_ms=56.0, max_passes=5):
    """Drop points whose removal makes the surrounding track plausible."""
    df = df.sort_values([id_col, "time"]).reset_index(drop=True).copy()
    removed_total = 0

    for step in range(max_passes):
        g = df.groupby(id_col)
        dt = g["time"].diff().dt.total_seconds()
        dist = np.hypot(g["x"].diff(), g["y"].diff())
        speed = dist / dt.replace(0, np.nan)

        bad = (speed > max_speed_ms).fillna(False)
        if not bad.any():
            print(f"  converged after {step} pass(es), "
                  f"{removed_total:,} points removed")
            break

        # a spike is fast in and fast out: drop the point between them
        spike = bad & bad.shift(-1, fill_value=False)
        drop = spike if spike.any() else bad
        removed_total += int(drop.sum())
        print(f"  pass {step + 1}: dropping {int(drop.sum()):,} "
              f"({'spikes' if spike.any() else 'endpoints'})")
        df = df[~drop].reset_index(drop=True)
    else:
        print(f"  did not converge in {max_passes} passes — "
              "the threshold may be too tight for this data")

    return df

Example 2 — measuring the cost of each threshold before choosing 

import numpy as np


def threshold_cost(df, id_col="track_id",
                   thresholds=(8, 14, 28, 56, 100)):
    """What each speed cap removes, and what it costs in distance."""
    g = df.groupby(id_col)
    dt = g["time"].diff().dt.total_seconds()
    dist = np.hypot(g["x"].diff(), g["y"].diff())
    speed = dist / dt.replace(0, np.nan)
    base = dist.sum()

    print(f"  unfiltered: {base / 1000:.3f} km, {len(df):,} points")
    for t in thresholds:
        keep = (speed <= t) | speed.isna()
        sub = df[keep]
        gg = sub.groupby(id_col)
        length = np.hypot(gg["x"].diff(), gg["y"].diff()).sum()
        print(f"  <= {t:3d} m/s ({t * 3.6:5.0f} km/h): "
              f"removed {int((~keep).sum()):5,} points, "
              f"{length / 1000:8.3f} km ({length / base - 1:+.2%})")

Run this before choosing. A threshold whose cost curve is flat is safe; one on a steep part of the curve is deleting movement.

Example 3 — smoothing that keeps the timestamps honest 

import numpy as np
import pandas as pd


def smooth_positions(df, id_col="track_id", window_s=5.0):
    """Time-weighted rolling mean, so irregular intervals do not distort it."""
    out = []
    for key, sub in df.groupby(id_col):
        sub = sub.sort_values("time").set_index("time")
        window = f"{int(window_s)}s"
        smoothed = sub[["x", "y"]].rolling(window, center=True,
                                           min_periods=1).mean()
        sub[["x_smooth", "y_smooth"]] = smoothed
        out.append(sub.reset_index())

    result = pd.concat(out, ignore_index=True)
    shift = np.hypot(result["x_smooth"] - result["x"],
                     result["y_smooth"] - result["y"])
    print(f"  smoothing moved points by a median of {shift.median():.2f} m, "
          f"p99 {shift.quantile(0.99):.2f} m")
    return result

A time-based rolling window rather than a row-based one is essential with irregular sampling: five rows can span five seconds or fifty. Note that smoothing reduces random jitter and cannot remove multipath bias — see GPS error explained.

Explanation

Why flagging beats deleting

A deleted row is a decision with no record. A flagged row lets a later step change its mind, lets a reviewer audit the choice, and lets you report how much of the data was affected.

Delete only at the end, in one place, and print the totals. The measured cost table above only exists because the filtering was expressed as flags that could be applied at several thresholds.

Why acceleration catches what speed misses

Consider a track at 1 m/s with one point displaced 200 m sideways. At 1-second sampling the speeds around it are 200 m/s in and 200 m/s out — both flagged. Filtering on speed alone might drop either neighbour.

The acceleration between them is roughly 400 m/s², which no vehicle produces. Pairing the two tests identifies the displaced point specifically.

A sustained offset — the receiver in a canyon reporting a parallel street for a minute — produces no spike at all. Every speed is plausible; only the route is wrong. That is invisible to any physical filter and needs map matching.

Why aggressive cleaning is not conservative

Removing data feels safe and is not. The measured cost of a 29 km/h cap was 333 points and 6.84% of distance on a dataset that genuinely contained vehicle travel.

The error is systematic rather than random: filters remove the fastest movement preferentially, so mean speeds fall, journey times rise, and mode classification shifts toward walking. A cleaning step that biases the result is worse than one that leaves some noise in.

Why the order of operations matters

Each step depends on the previous:

  1. Duplicates first — otherwise zero intervals produce infinities that break every subsequent computation.
  2. Identity second — otherwise derived speeds compare different journeys.
  3. Derived columns third — they are the evidence for everything after.
  4. Spike flags fourth, iterating.
  5. Gap splitting fifth — a gap is not a spike.
  6. Stop collapsing last — it changes distance, and you want to know the before and after.

Doing gap splitting before spike removal, for instance, means a spike at a leg boundary is never seen.

Deleting suspect GPS rows immediately against flagging them and deleting once at the end.
Flags make the threshold a measurable decision rather than an assumption.

Edge cases or notes

  • Drop exact duplicates first. 2.3% of a real download.
  • Guard dt == 0 before dividing.
  • Iterate spike removal; one spike can hide another.
  • Flag, do not delete, until the end.
  • Filter on impossibility, not implausibility, for mixed-mode data.
  • Use a time-based rolling window for smoothing, not a row-based one.
  • Smoothing does not remove multipath, only jitter.
  • Report what you removed, as a count and as a share of distance.

FAQ

How do I clean a GPS track in Python?

Drop exact duplicates, fix the track identity, derive speed and acceleration, flag physically impossible values, split at long gaps, and collapse stops — in that order.

What speed threshold should I use?

One justified by the mode of transport, and generous. On real traces a 200 km/h cap removed 13 points and 0.44% of distance; a 29 km/h cap removed 333 points and 6.84%.

Should I delete or flag bad points?

Flag, and delete once at the end. Flags are auditable and let you measure what each threshold costs before committing.

Why filter on acceleration as well as speed?

A single displaced point creates a high speed in and out. The acceleration between them identifies which point is wrong.

Does smoothing help?

It reduces random jitter. It cannot remove multipath bias, which is correlated — a smoothed track can be smoothly wrong.

Why did cleaning make my distances shorter?

Partly by removing spikes, which is intended, and partly by removing real fast movement if the threshold is too tight. Measure the cost curve before choosing.

Should I remove points with poor HDOP?

If the field is present, yes — it is direct evidence about the fix quality, better than any value inferred from the positions.