Why Are My Flow Totals Bigger Than the Flows on My Map?

Problem statement

You sum an origin–destination table and report "380,586 commuters", then draw the flows. The lines on the map add up to far less. The difference is the diagonal: people whose origin and destination are the same zone. They are in every OD table and they are real trips, but they are not flows between places and cannot be drawn as lines. Counting them as flows inflates every total. Worse, their share is not a property of the data. It depends on the size of the zones, so the same commuting data gives a different "flow total" at every level of aggregation.

This page measures the diagonal of Rhode Island's LODES commuting data at four zone sizes, from census blocks to counties, and separates internal trips from flows.

Quick answer

Split every OD total into internal (origin = destination) and between-zone parts, and report both:

def split(level):
    o, d = od.h_geocode.str[:WIDTH[level]], od.w_geocode.str[:WIDTH[level]]
    internal = od.S000[o == d].sum()
    return internal, od.S000.sum() - internal

for level in WIDTH:
    i, b = split(level)
    print(f"{level:7s}: internal {i:>7,} ({i / (i + b):5.1%}), between zones {b:>7,}")

block  : internal     302 ( 0.1%), between zones 380,284
tract  : internal  19,888 ( 5.2%), between zones 360,698
county : internal 244,279 (64.2%), between zones 136,307

See it in a total, watch it with zone size, keep it separate

1. See the problem in a total

Summing the whole OD table counts every job, including those held in the worker's own town. Only the between-town part can become lines:

towns = od.assign(h=od.h_geocode.map(TOWN), w=od.w_geocode.map(TOWN))
total, between = towns.S000.sum(), towns.loc[towns.h != towns.w, "S000"].sum()
print(f"all jobs {total:,}; between towns {between:,}; 'flows' over-reported by {total / between - 1:.0%} if the diagonal is kept")

all jobs 380,586; between towns 292,363; 'flows' over-reported by 30% if the diagonal is kept

2. Watch it change with zone size

At block level almost everyone commutes "between zones", because blocks are tiny. At county level a large share stays inside. The diagonal share grows with zone size, so a flow total means nothing without its zones:

shares = {lvl: split(lvl)[0] / od.S000.sum() for lvl in WIDTH}
print(", ".join(f"{k}: {v:.0%}" for k, v in shares.items()))
print(f"zones: {', '.join(f'{k} {od.h_geocode.str[:WIDTH[k]].nunique():,}' for k in WIDTH)}")

block: 0%, tract: 5%, county: 64%
zones: block 20,989, tract 247, county 5

3. Keep the diagonal, but as its own measure

The internal share is useful in its own right: as self-containment it says how local each place is. Report it per zone next to the flows:

self_cont = towns[towns.h == towns.w].groupby("h").S000.sum() / towns.groupby("h").S000.sum()
print(f"town self-containment: median {self_cont.median():.0%}, range {self_cont.min():.0%}–{self_cont.max():.0%}")

town self-containment: median 18%, range 8%–76%

A flow table that carries both parts

The data used on this page

LODES version 8 for Rhode Island, 2021 (main file, all jobs) and its block crosswalk for town assignments. Block codes contain their tract (first 11 characters) and county (first 5).

from pathlib import Path

import pandas as pd
import requests

Path("data").mkdir(exist_ok=True)
L = "https://lehd.ces.census.gov/data/lodes/LODES8/ri/"
for name, url in {"ri_od_main_JT00_2021.csv.gz": L + "od/ri_od_main_JT00_2021.csv.gz", "ri_xwalk.csv.gz": L + "ri_xwalk.csv.gz"}.items():
    if not Path("data", name).exists():
        Path("data", name).write_bytes(requests.get(url, timeout=600).content)
od = pd.read_csv("data/ri_od_main_JT00_2021.csv.gz", dtype={"w_geocode": str, "h_geocode": str})
TOWN = pd.read_csv("data/ri_xwalk.csv.gz", dtype=str, usecols=["tabblk2020", "ctycsub"]).set_index("tabblk2020").ctycsub
WIDTH = {"block": 15, "tract": 11, "county": 5}
print(f"{len(od):,} block pairs, {od.S000.sum():,} jobs")

360,924 block pairs, 380,586 jobs

Towns are not a prefix of the block code, so they are joined through the crosswalk rather than sliced; the quick answer covers the three levels that are.

Example 1: a flow table that carries both parts

summary = (towns.assign(kind=lambda d: (d.h == d.w).map({True: "internal", False: "between towns"}))
                .groupby("kind").S000.sum())
print(summary.to_string())

kind
between towns    292363
internal          88223

Every table has a diagonal to decide about

Every OD table has a diagonal, and every flow total has to decide what to do with it. The diagonal holds trips that start and end in the same zone: people who live and work in the same town, moves within a county, bike trips that end at the starting dock. They are real, and often the largest single group, but they have no length and no direction between places. A flow map cannot show them, and a total of "flows" that includes them describes the zones as much as the movement.

The dependence on zone size is the trap. Rhode Island's commuting data has the same jobs at every level, but the internal share runs from almost nothing between blocks to over half within counties. Two analyses at different levels can report very different "flow totals" from identical data. The fix is bookkeeping: always split totals into internal and between-zone parts, name the zones, and map only the between-zone part as flows.

Migration, bike share and self-containment

• Migration data often excludes moves within the same county by design; check the definition before adding a diagonal.
• Bike share round trips are the diagonal of a station OD table; they are frequent at leisure locations.
• Matrix plots dominated by the diagonal need a log colour scale or the diagonal masked.
• Self-containment and the share of jobs filled locally are different ratios (row vs column share); name which one you report.

Internal links

• How to build an origin–destination matrix — reading the diagonal
• How to map commuting flows from LODES — the town flows

FAQ

What are self-flows in an origin–destination matrix?

Flows whose origin and destination are the same zone: the diagonal of the matrix. They are internal trips, not movement between zones.

Should I include internal trips in flow totals?

Report them, but separately. A flow total that mixes internal and between-zone trips cannot be compared across zone systems.

Why does the share of internal trips change with zone size?

Larger zones contain more of each trip's origin and destination, so more trips start and end inside one zone.

How do I remove the diagonal in pandas?

Filter rows where origin equals destination (df[df.o != df.d]), or set the diagonal of a pivoted matrix to zero with np.fill_diagonal.