⚙️ How-To Coastal & marine Movement & time Coastal & Marine · step 13 of 20 Updated

How to clean AIS vessel tracks in Python

Problem statement

AIS is a self-reported broadcast from a transponder, and everything that can be typed in by hand is sometimes wrong. Positions are usually good; identities, speeds, headings and destinations are not. A raw AIS extract cleaned only by deduplication produces vessel tracks that teleport across oceans, ships that are in two places at once, and speeds of several hundred knots.

The cleaning rules are the same as for any GPS trajectory, with four additions specific to AIS: the MMSI is not a reliable vessel identifier, the sentinel values are meaningful, the message types carry different fields, and receiver coverage gaps look exactly like a vessel going dark.

This guide builds the cleaning pipeline and the checks that show what each rule removed.

Quick answer

Clean in this order — sentinels, coordinates, then per-vessel kinematics:

import numpy as np, pandas as pd

SENTINELS = {"lat": 91.0, "lon": 181.0, "sog": 102.3, "cog": 360.0, "heading": 511}

def clean_ais(df):
    n0 = len(df)
    df = df[(df.lat.abs() <= 90) & (df.lon.abs() <= 180)]
    df = df[(df.lat != SENTINELS["lat"]) & (df.lon != SENTINELS["lon"])]
    df.loc[df.sog >= SENTINELS["sog"], "sog"] = np.nan
    df.loc[df.cog >= SENTINELS["cog"], "cog"] = np.nan
    df.loc[df.heading >= SENTINELS["heading"], "heading"] = np.nan
    df = df.sort_values(["mmsi", "timestamp"]).drop_duplicates(["mmsi", "timestamp"])
    print(f"{n0:,} → {len(df):,} rows after sentinel and coordinate filtering")
    return df

The AIS sentinels are specified values, not errors: latitude 91 and longitude 181 mean "position not available", speed over ground 102.3 means "not available", course 360 means "not available" and heading 511 means "not available". Treating them as data is the first thing that goes wrong.

Flow from raw AIS messages through sentinel filtering, identity checks, kinematic filtering and gap splitting to clean tracks.
Five stages; the identity stage is the one that has no equivalent in GPS cleaning.

Step-by-step solution

1. Understand the message types

Position reports (types 1, 2, 3 for Class A; 18, 19 for Class B) carry position, speed and course. Static and voyage data (type 5 for Class A, 24 for Class B) carries the name, callsign, dimensions, draught and destination. Joining them means a join on MMSI, and the static data changes over a voyage.

2. Remove sentinel values before anything numeric

A mean speed computed with 102.3 knots in it is meaningless, and a bounding box computed with latitude 91 is impossible.

3. Treat the MMSI as a weak identifier

MMSIs are mistyped, shared between vessels, reused after reassignment, and occasionally set to obvious placeholders. Check for the same MMSI reporting from impossible pairs of positions, and treat a vessel whose reported name or dimensions change mid-track as suspect.

4. Filter by implied speed, not by reported speed

The reported speed over ground can be wrong while the positions are fine, and vice versa. Computing the speed implied by consecutive positions and times finds the teleports; comparing it with the reported speed finds the transponder problems.

5. Split tracks at gaps rather than interpolating across them

A vessel outside receiver range produces a gap that looks like a straight line at moderate speed if you join across it. Splitting into segments at a time or distance threshold keeps the uncertainty visible.

6. Resample only within segments

Once the track is split, a regular resample inside each segment is safe. Across a gap it invents a voyage.

7. Report what each rule removed

The counts are the result. A pipeline that silently drops 40% of the messages has made a decision nobody saw.

Table of AIS sentinel values for position, speed, course and heading and what each means.
Four specified "not available" values that are valid numbers in the wrong hands.

Code examples

Example 1 — implied speed and the teleport filter 

import numpy as np, pandas as pd
from pyproj import Geod

geod = Geod(ellps="WGS84")
MAX_KNOTS = 40.0          # above any merchant vessel; adjust for the fleet

def kinematics(df):
    out = []
    for mmsi, g in df.sort_values("timestamp").groupby("mmsi", sort=False):
        g = g.copy()
        if len(g) < 2:
            g["implied_kn"] = np.nan
            out.append(g)
            continue
        lon, lat = g.lon.values, g.lat.values
        _, _, dist = geod.inv(lon[:-1], lat[:-1], lon[1:], lat[1:])
        dt = np.diff(g.timestamp.values).astype("timedelta64[s]").astype(float)
        with np.errstate(divide="ignore", invalid="ignore"):
            speed = np.r_[np.nan, dist / np.where(dt > 0, dt, np.nan) * 1.94384]
        g["implied_kn"] = speed
        g["gap_s"] = np.r_[np.nan, dt]
        out.append(g)
    return pd.concat(out)

k = kinematics(clean)
bad = k.implied_kn > MAX_KNOTS
print(f"implied speed over {MAX_KNOTS} kn: {bad.sum():,} of {len(k):,} "
      f"({bad.mean():.2%}) on {k.loc[bad, 'mmsi'].nunique():,} MMSIs")
print(k.implied_kn.describe(percentiles=[.5, .9, .99, .999]).round(2))

Report the percentiles before choosing the threshold. A fleet of fast ferries and a fleet of bulk carriers need different numbers, and the 99.9th percentile tells you where the tail starts.

Example 2 — split into segments at gaps 

import numpy as np, pandas as pd

def segment(df, max_gap_s=3600, max_gap_nm=50):
    out = []
    for mmsi, g in df.groupby("mmsi", sort=False):
        g = g.sort_values("timestamp").copy()
        gap_time = g["gap_s"].fillna(np.inf) > max_gap_s
        gap_dist = (g["implied_kn"].fillna(0) * g["gap_s"].fillna(0) / 3600) > max_gap_nm
        g["segment"] = (gap_time | gap_dist).cumsum()
        out.append(g)
    seg = pd.concat(out)
    sizes = seg.groupby(["mmsi", "segment"]).size()
    print(f"{len(sizes):,} segments from {seg.mmsi.nunique():,} vessels; "
          f"median {sizes.median():.0f} points, "
          f"{(sizes < 3).sum():,} segments with fewer than 3 points")
    return seg

Segments with one or two points are not tracks; count them and drop them explicitly rather than letting a line-building step produce degenerate geometry.

Example 3 — the identity checks 

import numpy as np, pandas as pd
from pyproj import Geod

geod = Geod(ellps="WGS84")

def identity_flags(df, static=None, max_knots=40.0):
    flags = []
    for mmsi, g in df.groupby("mmsi", sort=False):
        g = g.sort_values("timestamp")
        row = {"mmsi": mmsi, "messages": len(g),
               "span_h": (g.timestamp.max() - g.timestamp.min()).total_seconds() / 3600}
        row["impossible_jumps"] = int((g["implied_kn"] > max_knots).sum())
        # a valid MMSI is nine digits and does not start with 0
        s = str(mmsi)
        row["malformed_mmsi"] = not (len(s) == 9 and s[0] != "0")
        if static is not None:
            st = static[static.mmsi == mmsi]
            row["distinct_names"] = st["shipname"].nunique()
            row["distinct_lengths"] = st["length_m"].nunique()
        flags.append(row)
    f = pd.DataFrame(flags)
    f["suspect"] = (f.impossible_jumps > 0) | f.malformed_mmsi | \
                   (f.get("distinct_names", 0) > 1)
    return f

A vessel broadcasting two different names on one MMSI within a day is either a data-entry error or two vessels sharing an identifier. Either way the track is not one vessel's, and joining a name to it produces a confident fiction.

Explanation

Why AIS positions are good and everything else is not

Position, speed over ground and course over ground come from the vessel's GNSS receiver and are as good as that receiver. Name, callsign, dimensions, draught, destination and navigational status are entered by the crew through a keypad, and they are entered rarely, copied between voyages and frequently left at whatever the last crew set. Analysis that depends on the typed fields needs independent verification; analysis that depends on position mostly does not.

Why implied speed beats reported speed for filtering

A transponder reporting a stuck speed while the positions move is a different fault from positions jumping while the speed is sensible. Computing the speed from consecutive positions and times catches the first; comparing the two catches the second. Using only the reported field misses both.

Why receiver coverage looks like going dark

Terrestrial AIS receivers have a range of tens of nautical miles and satellite AIS has revisit gaps and message collisions in busy areas. A vessel that disappears for six hours in mid-ocean has almost certainly just left coverage. Distinguishing that from a deliberate transponder switch-off needs the coverage footprint of the receiving network, which most extracts do not ship — so a gap is evidence of a gap and nothing more.

Why the sentinel values matter more than they should

They are valid numbers in the message's range, so nothing rejects them. Latitude 91 places a vessel beyond the pole, speed 102.3 knots is faster than any ship, and heading 511 is an impossible bearing. Any aggregation computed before they are removed carries them, and a mean speed that includes a handful of 102.3s is visibly wrong only if somebody looks.

Two panels separating what an AIS coverage gap tells you from what it does not, including that it is not evidence of a transponder being switched off.
Without a coverage footprint, "went dark" is an inference the data does not support.

Edge cases or notes

  • Class A and Class B differ. Reporting intervals, fields and power are not the same.
  • Timestamps may be receiver time, not vessel time. Check which.
  • Duplicate messages are normal where receiver footprints overlap.
  • Anchored vessels drift. A stationary filter on speed alone keeps swinging at anchor.
  • The antimeridian breaks implied speed if longitudes are differenced naively.
  • Base stations and aids to navigation also broadcast and are not vessels.
  • Draught changes with cargo. It is not a constant per vessel.
  • Report every rule's count. The filtering is the analysis.

FAQ

What are the AIS sentinel values?

Latitude 91 and longitude 181 mean the position is unavailable, speed over ground 102.3 means the speed is unavailable, course 360 and heading 511 mean those are unavailable. All are valid numbers in the message format.

Is the MMSI a reliable vessel identifier?

No. MMSIs are mistyped, shared, reused and sometimes set to placeholders. Check for impossible jumps and for a single MMSI reporting different names or dimensions.

Should I filter on reported or implied speed?

Implied, computed from consecutive positions and times. Compare it with the reported speed to find transponder faults.

What do I do about long gaps?

Split the track into segments rather than interpolating. A gap in mid-ocean is usually the edge of receiver coverage, not a vessel going dark.

Why do I have duplicate messages?

Because overlapping receiver footprints report the same broadcast. Deduplicate on MMSI and timestamp.

What speed threshold should I use?

Look at the percentiles of implied speed for your fleet first. Forty knots is above any merchant vessel and too low for some fast ferries.