Match Quality Explained: Reading a Geocoder's Confidence Score

Problem statement

Every geocoder returns a number alongside the coordinate — importance, confidence, score, match_code, relevance — and every team eventually adopts a threshold: keep everything above 0.7, discard the rest.

The threshold is almost always wrong, for three reasons:

• The scores are not comparable between providers, or even between query types on the same provider.
• The score measures how well the text matched, not how close the coordinate is.
• The distribution is not uniform. A 0.7 cut can discard nothing or half the file, depending on the query mix.

Measured on Nominatim: the country "United Kingdom" scored 0.9389 and the actual building at 10 Downing Street scored 0.5506 (Example 3). A 0.7 threshold keeps the useless answer and throws away the exact one.

Quick answer

Never filter on the score alone. Filter on precision, use the score to detect ambiguity, and use geometry to validate:

def accept(hits, worst_precision="road", min_gap=0.05):
    """Three independent tests, all of which must pass."""
    if not hits:
        return None, "no_match"

    top = hits[0]
    level = top.get("addresstype")
    if PRECISION_RANK.get(level, 9) > PRECISION_RANK[worst_precision]:
        return None, f"too_coarse:{level}"          # test 1: geometry precision

    if len(hits) > 1:
        gap = float(top.get("importance", 0)) - float(hits[1].get("importance", 0))
        if gap < min_gap:
            return None, f"ambiguous:gap={gap:.3f}"  # test 2: score separation

    return top, "ok"                                 # test 3 is spatial, later


PRECISION_RANK = {"house": 0, "building": 0, "office": 0, "amenity": 0,
                  "road": 1, "postcode": 2, "suburb": 3, "city": 4,
                  "state": 5, "country": 6}

for query in ("10 Downing Street, London", "United Kingdom", "Springfield", "San Antonio"):
    top, reason = accept(geocoder.search(query, limit=3), worst_precision="city")
    print(f"{query:28} {reason:22} {top['display_name'][:40] if top else ''}")

10 Downing Street, London    ok                     10 Downing Street, 10, Downing Street, W
United Kingdom               too_coarse:country     
Springfield                  ambiguous:gap=0.001    
San Antonio                  ok                     San Antonio, Bexar County, Texas, United

The second test is the one people skip and the one that catches the most damage: a top result that barely beat the runner-up is a coin flip the geocoder made on your behalf.

What the score measures, precision apart from confidence

1. Find out what the score actually measures

Read the provider's definition before using the number, because the same field name means different things:

• String similarity — how closely the returned address text matches the query. Useful; comparable within one provider.
• Prominence or importance — how notable the matched feature is. Nominatim's importance is this, derived partly from Wikipedia linkage. It says nothing about your query.
• Composite quality — a blend of match completeness, reference-data quality and geometry precision, usually undocumented in detail.
• Categorical match codes — not a number at all: Exact, UpHierarchy, Ambiguous. These are the most honest, because they cannot be averaged.

The single most useful sentence in any geocoding API's documentation is the one defining this field. If it does not exist, treat the number as a ranking key and nothing more.

2. Separate precision from confidence, permanently

They fail independently, so they need separate columns:

Only the top-left cell is safe to use unexamined. The top-right is the dangerous one, because it looks precise, and the bottom-left is the one that produces neat maps of wrong places.

3. Use the gap between candidates as the ambiguity signal

Ask for three results instead of one. The absolute score is not comparable across queries; the difference between the top two is, because both were scored by the same function on the same query.

A large gap means the geocoder had a clear winner. A small gap means it picked one of several equals — and with limit=1 you would never have known. Place names are duplicated on a scale that makes this common: among the GeoNames gazetteer's populated places, 132,552 distinct names occur in more than one country, San Antonio names 2,384 places across 25 countries, and Springfield names 68 places in the United States alone (Example 4). Nominatim's top two answers for "Springfield" — Illinois and Massachusetts — were 0.0015 apart.

4. Calibrate the threshold on labelled data, once

If you are going to use a numeric cut, earn it: take 200 addresses whose true locations you know — from a previous verified geocode, a survey, or a manual check — and plot the error distance against the score.

You will usually find the relationship is weak and steppy rather than smooth: the score separates "found the right kind of thing" from "did not", and within each precision level it carries almost no information about distance. That is a finding, not a failure — it tells you the threshold belongs on the precision field. Example 2 does this for the food businesses of Ledbury.

5. Validate spatially, because that is the test that does not lie

Whatever the score says, the coordinate can be checked against something you already believe: the region the record claims to be in, the postcode polygon, the country boundary, a previous geocode of the same address.

The spatial test is independent of the provider's opinion, which is exactly what makes it worth more than the provider's opinion.

A quality record instead of a number

The data used on this page

Live answers from the public Nominatim service, at its limit of one request a second and cached to disk so a re-run repeats nothing; the food businesses of Ledbury, Herefordshire, from the Food Standards Agency register, whose coordinates serve as the labels; and the GeoNames allCountries gazetteer, the same file the spatial join guide prepares. The register is a copy of the API's answer saved on 10 October 2026, so it stays the one measured here.

import json
import math
import os
import time
import zipfile

import duckdb
import pandas as pd
import requests


def haversine_m(a, b):
    (lat1, lon1), (lat2, lon2) = a, b
    p1, p2 = math.radians(lat1), math.radians(lat2)
    h = (math.sin((p2 - p1) / 2) ** 2
         + math.cos(p1) * math.cos(p2) * math.sin(math.radians(lon2 - lon1) / 2) ** 2)
    return 2 * 6_371_008.8 * math.asin(math.sqrt(h))


class Nominatim:
    """One request a second, identified, and cached: the terms of use, plus repeatability."""
    def __init__(self, cache="nominatim_cache.json"):
        self.cache_path, self.last = cache, 0.0
        self.cache = json.load(open(cache)) if os.path.exists(cache) else {}
        self.session = requests.Session()
        self.session.headers["User-Agent"] = "spatialworkflow.io guide (geocoder match quality)"

    def search(self, query, limit=3):
        key = f"{limit}|{query}"
        if key not in self.cache:
            time.sleep(max(0.0, 1.05 - (time.time() - self.last)))
            reply = self.session.get("https://nominatim.openstreetmap.org/search", timeout=60,
                                     params={"q": query, "format": "jsonv2", "limit": limit})
            self.last = time.time()
            self.cache[key] = reply.json()
            json.dump(self.cache, open(self.cache_path, "w"))
        return self.cache[key]


geocoder = Nominatim()

if not os.path.exists("ledbury.csv"):
    # live: requests.get("https://api.ratings.food.gov.uk/Establishments", headers={"x-api-version": "2"},
    #                    params={"localAuthorityId": 383, "pageSize": 5000}, timeout=300)   # Herefordshire
    # read instead as the API answered on 10 October 2026, so the register stays put
    est = pd.DataFrame(requests.get("https://data.spatialworkflow.io/fhrs-api-383-2026-10-10.json", timeout=300).json()["establishments"])
    est = est[est["AddressLine4"].eq("Ledbury")].copy()
    est["lat"] = est["geocode"].map(lambda g: float(g["latitude"]) if g and g.get("latitude") else None)
    est["lon"] = est["geocode"].map(lambda g: float(g["longitude"]) if g and g.get("longitude") else None)
    lines = ["AddressLine1", "AddressLine2", "AddressLine3", "AddressLine4", "PostCode"]
    est["query"] = est[lines].apply(lambda r: ", ".join(x for x in r if x), axis=1)
    est.dropna(subset=["lat", "lon"])[["query", "lat", "lon"]].to_csv("ledbury.csv", index=False)
ledbury = list(pd.read_csv("ledbury.csv").itertuples(index=False, name=None))

if not os.path.exists("places.parquet"):
    with requests.get("https://download.geonames.org/export/dump/allCountries.zip", stream=True,
                      timeout=600) as reply:
        with open("allCountries.zip", "wb") as out:
            for chunk in reply.iter_content(1 << 20):
                out.write(chunk)
    zipfile.ZipFile("allCountries.zip").extract("allCountries.txt")
    duckdb.sql("""copy (select column01 as name, column06 as feature_class, column08 as country
                        from read_csv('allCountries.txt', delim = '\t', header = false, quote = '',
                                      sample_size = -1))
                  to 'places.parquet' (format parquet)""")
print(len(ledbury), "labelled Ledbury addresses; GeoNames rows:",
      duckdb.sql("select count(*) from 'places.parquet'").fetchone()[0])

78 labelled Ledbury addresses; GeoNames rows: 13472324

Example 1 — a quality record, not a quality number

from dataclasses import dataclass


@dataclass
class MatchQuality:
    precision: str              # what kind of feature matched
    score: float | None         # the provider's number, unmodified
    score_gap: float | None     # top score minus runner-up
    candidates: int             # how many were returned
    bbox_span_km: float | None  # how big the matched feature is
    text_similarity: float      # our own comparison, provider-independent

    @property
    def verdict(self) -> str:
        if self.precision in ("country", "state", "region"):
            return "unusable"
        if self.score_gap is not None and self.score_gap < 0.05:
            return "ambiguous"
        if self.text_similarity < 0.6:
            return "suspect"
        if self.precision in ("house", "building", "office"):
            return "good"
        return "coarse"


def normalise(text: str) -> str:
    """Lower case, punctuation to spaces, one space between words."""
    import re
    return re.sub(r"\s+", " ", re.sub(r"[^\w\s]", " ", text.lower())).strip()


def text_similarity(query: str, matched: str) -> float:
    """A provider-independent second opinion: the share of query words in the answer.

    Not an edit-distance ratio: Nominatim's display_name carries the whole address
    hierarchy, and a ratio against it punishes a perfect match for being long.
    """
    asked, found = set(normalise(query).split()), set(normalise(matched).split())
    return len(asked & found) / len(asked) if asked else 0.0


for query in ("10 Downing Street, London", "United Kingdom", "Springfield"):
    hits = geocoder.search(query, limit=3)
    top = hits[0]
    gap = float(top["importance"]) - float(hits[1]["importance"]) if len(hits) > 1 else None
    south, north, west, east = map(float, top["boundingbox"])
    q = MatchQuality(precision=top["addresstype"], score=float(top["importance"]), score_gap=gap,
                     candidates=len(hits), bbox_span_km=round(haversine_m((south, west), (north, east)) / 1000, 2),
                     text_similarity=round(text_similarity(query, top["display_name"]), 2))
    print(f"{query:28} {q.verdict:9} {q}")

10 Downing Street, London    good      MatchQuality(precision='office', score=0.5505744539169398, score_gap=None, candidates=1, bbox_span_km=0.05, text_similarity=1.0)
United Kingdom               unusable  MatchQuality(precision='country', score=0.9388533772580386, score_gap=None, candidates=1, bbox_span_km=1615.49, text_similarity=1.0)
Springfield                  ambiguous MatchQuality(precision='city', score=0.6126046344830106, score_gap=0.0014626072008783186, candidates=3, bbox_span_km=30.12, text_similarity=1.0)

The first version of this function used difflib's ratio against the whole display name, and called 10 Downing Street "suspect" at 0.32 — because "10 Downing Street, 10, Downing Street, Westminster, Millbank, London, …" is much longer than the query. Measure whether the answer contains what you asked for, not whether the two strings look alike. Computing your own text similarity is worth the four lines: it is the one quality signal that means the same thing across every provider, which makes multi-provider tables comparable.

Example 2 — measuring whether the score predicts anything

import statistics


def calibrate(labelled, geocoder):
    """labelled: [(query, true_lat, true_lon), ...] with verified truth."""
    buckets, missed = {}, 0
    for query, tlat, tlon in labelled:
        hits = geocoder.search(query)
        if not hits:
            missed += 1
            continue
        top = hits[0]
        err = haversine_m((tlat, tlon), (float(top["lat"]), float(top["lon"])))
        band = round(float(top.get("importance", 0)) * 10) / 10
        buckets.setdefault(band, []).append(err)

    print(f"{len(labelled)} queries, {missed} with no match")
    print(f"{'score':>6} {'n':>5} {'median err':>12} {'90th pct':>10}")
    for band in sorted(buckets):
        errs = sorted(e for e in buckets[band] if e is not None)
        if not errs:
            continue
        print(f"{band:6.1f} {len(errs):5d} {statistics.median(errs):10,.0f} m "
              f"{errs[int(0.9 * len(errs)) - 1]:8,.0f} m")


calibrate(ledbury, geocoder)

from collections import Counter
kinds = {}
for query, _, _ in ledbury:
    hits = geocoder.search(query)
    if hits:
        band = round(float(hits[0]["importance"]) * 10) / 10
        kinds.setdefault(band, Counter())[hits[0]["addresstype"]] += 1
print({band: dict(counts) for band, counts in sorted(kinds.items())})

78 queries, 30 with no match
 score     n   median err   90th pct
   0.0    38         36 m       85 m
   0.1    10         78 m      711 m
{0.0: {'amenity': 20, 'building': 5, 'shop': 11, 'tourism': 2}, 0.1: {'road': 10}}

The labels here are the Food Standards Agency's own coordinates for 78 food businesses in Ledbury, which are good to tens of metres rather than to the doorstep, so errors below that are noise. Two things stand out. Nominatim found nothing for a large share of full address strings — typical of free-text queries with shop names in them. And among the matches, the higher score band had the larger errors, for a reason the last line shows: all 10 answers in it were roads — Nominatim falling back to the street when it could not find the business — while every building, shop and amenity it did find scored lower. A threshold on this score would have kept the fallbacks and dropped the hits.

Run this once per provider. If the median error is flat across score bands, the score is a ranking key, not a quality measure — and every threshold argument you were about to have is settled.

Example 3 — the ambiguity check that actually pays

def ambiguity_report(queries, geocoder, gap_threshold=0.05):
    """How much of this file is the geocoder guessing at?"""
    guessed, clear, missing = [], 0, 0
    for q in queries:
        hits = geocoder.search(q, limit=3)
        if not hits:
            missing += 1
            continue
        if len(hits) == 1:
            clear += 1
            continue
        gap = float(hits[0].get("importance", 0)) - float(hits[1].get("importance", 0))
        if gap < gap_threshold:
            apart_km = haversine_m((float(hits[0]["lat"]), float(hits[0]["lon"])),
                                   (float(hits[1]["lat"]), float(hits[1]["lon"]))) / 1000
            guessed.append((q, hits[0]["display_name"], hits[1]["display_name"], round(gap, 4),
                            round(apart_km, 1)))
        else:
            clear += 1

    print(f"clear   {clear:5d}")
    print(f"guessed {len(guessed):5d}   <- these need a human or more context")
    print(f"missing {missing:5d}")
    for q, a, b, gap, km in guessed[:5]:
        print(f"  {q!r}: gap {gap}, {km:,} km apart\n    {a[:60]}\n    vs {b[:60]}")
    return guessed


places = ["Springfield", "Newport", "Richmond", "Kingston", "San Antonio", "Portland",
          "Cambridge", "Dublin", "Paris", "10 Downing Street, London", "United Kingdom"]
guessed = ambiguity_report(places, geocoder)

clear       7
guessed     4   <- these need a human or more context
missing     0
  'Springfield': gap 0.0015, 1,452.5 km apart
    Springfield, Sangamon County, Illinois, United States
    vs Springfield, Hampden County, Massachusetts, United States
  'Newport': gap 0.0038, 5,148.7 km apart
    Newport, Newport County, Rhode Island, 02840, United States
    vs Newport, Cymru / Wales, United Kingdom
  'Cambridge': gap 0.0, 1.7 km apart
    Cambridge, Cambridgeshire, Cambridgeshire and Peterborough, 
    vs Cambridge, Cambridgeshire, Cambridgeshire and Peterborough, 
  'Paris': gap 0.0, 2.2 km apart
    Paris, Île-de-France, France métropolitaine, France
    vs Paris, Île-de-France, France métropolitaine, France

Example 4 — how often do place names repeat?

import duckdb

con = duckdb.connect()
populated = "(select * from 'places.parquet' where feature_class = 'P')"
print(con.execute(f"""select count(*) from (select name from {populated}
                      group by name having count(distinct country) > 1)""").fetchone()[0],
      "populated-place names occur in more than one country")
for name in ("San Antonio", "Springfield"):
    n, countries, us = con.execute(f"""select count(*), count(distinct country),
                                       count(*) filter (where country = 'US')
                                       from {populated} where name = '{name}'""").fetchone()
    print(f"{name}: {n:,} places in {countries} countries, {us} in the United States")

132552 populated-place names occur in more than one country
San Antonio: 2,384 places in 25 countries, 8 in the United States
Springfield: 121 places in 12 countries, 68 in the United States

The output of this function is the most useful thing you can hand a domain expert: a short list of specific decisions the machine could not make, rather than a whole file to check. Read it before acting on it, though. Springfield and Newport are genuine coin flips between places an ocean apart. Cambridge and Paris are not: their gap is exactly 0 because OpenStreetMap holds each city more than once — Cambridge as a place node and an administrative boundary 1.7 km apart, Paris as two administrative boundaries 2.2 km apart. A gap test needs a distance test beside it: two near-tied candidates a couple of kilometres apart are one place, while Springfield's are 1,452 km apart and Newport's 5,149.

Why scores are not comparable between providers

Why scores are not comparable between providers

Each provider computes its number from a different mixture of ingredients — string distance, feature prominence, reference-data confidence, population, query completeness — and normalises it into a different range with a different shape.

A 0.8 from one provider and a 0.8 from another have no relationship whatever. This matters in practice because multi-provider pipelines are normal: one geocoder for the domestic file, another for the international rows. Any threshold has to be per provider, and any comparison has to go through a signal you computed yourself.

Why prominence-based scores invert the ranking you want

Nominatim's importance measures notability. That is the right ranking for a search box — someone typing "London" almost always means the big one — and the wrong ranking for a batch of addresses, where fame is uncorrelated with correctness.

The measured example is stark: the country of the United Kingdom scores 0.9389 and the specific building you asked for scores 0.5506. Sorting a file by that score puts the least useful answers at the top.

Why the score cannot know how far away the coordinate is

The geocoder scores the match between two pieces of text. The distance error comes from the reference geometry — whether it stores that building as a point, interpolates it along a street segment, or has only the postal area.

Those two things are almost independent. A perfect text match against a street-interpolated reference is a confident answer that is 40 m out; a mediocre text match against an address-point reference is an uncertain answer that is exact. This is why precision has to be a separate field and why the calibration in Example 2 so often comes out flat.

Why "no match" is a quality signal, not a failure

A geocoder that returns nothing has told you something true. One that returns a city centroid for the same query has told you something false in a format that looks identical to a success.

Providers differ in how willing they are to say nothing, and this is worth testing during selection. A higher no-match rate with honest reporting is generally cheaper to work with than a 100% hit rate padded with fallbacks — the first produces a work queue, the second produces a map that is quietly wrong.

Several candidates, stored scores and counts

• Ask for more than one candidate during development, even if production uses one. The gap is free information.
• Store the score, the precision and the candidate count. Any one of them alone is insufficient.
• Scores drift between provider versions. A threshold calibrated last year is not calibrated now.
• Empty queries and single-token queries produce high scores against countries and regions; filter them before sending.
• Structured queries usually score differently from free text on the same provider — calibrate the mode you use.
• A limit=1 request hides the ambiguity entirely and is the single most common cause of confident wrong answers.
• Match codes beat numbers. If a provider offers categorical codes, prefer them; they cannot be averaged into nonsense.
• Do not average scores across a file as a quality metric. Report the distribution by precision level instead.

Internal links

• Geocoding explained: from an address string to a coordinate — the pipeline the score comes out of
• Rooftop, interpolated or centroid: geocoding precision levels — the field to filter on instead
• How to validate geocoding results before you trust them — the spatial tests
• Choosing a geocoder: coverage, licence and cost compared — comparing providers on more than the score
• Fixing a geocoder that matches the wrong town — the ambiguity failure in practice
• Address matching explained: why exact string equality fails — computing your own similarity
• How to geocode with Nominatim from Python without being blocked — where importance comes from
• Spatial data quality dimensions — quality as a set of measurable properties

FAQ

What confidence threshold should I use?

None, on its own. Filter on the precision level, flag results where the top two candidates are within about 0.05 of each other, and validate the coordinate against a boundary you trust.

Why does a country score higher than the exact address?

Because some scores measure prominence rather than match quality. Nominatim's importance gave the United Kingdom 0.9389 and the building at 10 Downing Street 0.5506.

Can I compare confidence scores between two geocoders?

No. Each provider computes and normalises the number differently. Compare a signal you compute yourself, such as string similarity between the query and the returned address.

What does a small gap between the top two results mean?

That the geocoder picked one of several equally good candidates. With 132,552 populated-place names occurring in more than one country, that is common — and with limit=1 it is invisible.

Does a high score mean the coordinate is accurate?

No. The score is about the text match; the distance error comes from the reference geometry. Calibrate on labelled data and you will usually find the relationship is weak within a precision level.

Is a no-match better than a low-confidence match?

Usually yes. A no-match produces a work queue; a low-confidence fallback produces a coordinate that looks like every other coordinate in the column.