Match Quality Explained: Reading a Geocoder's Confidence Score #

Problem statement #

Every geocoder returns a number alongside the coordinate — importance, confidence, score, match_code, relevance — and every team eventually adopts a threshold: keep everything above 0.7, discard the rest.

The threshold is almost always wrong, for three reasons:

• The scores are not comparable between providers, or even between query types on the same provider.
• The score measures how well the text matched, not how close the coordinate is.
• The distribution is not uniform. A 0.7 cut can discard nothing or half the file, depending on the query mix.

Measured on one provider: the country "United Kingdom" scored 0.9389 and the actual building at 10 Downing Street scored 0.5506. A 0.7 threshold keeps the useless answer and throws away the exact one.

Quick answer #

Never filter on the score alone. Filter on precision, use the score to detect ambiguity, and use geometry to validate:

def accept(hits, worst_precision="road", min_gap=0.05):
    """Three independent tests, all of which must pass."""
    if not hits:
        return None, "no_match"

    top = hits[0]
    level = top.get("addresstype")
    if PRECISION_RANK.get(level, 9) > PRECISION_RANK[worst_precision]:
        return None, f"too_coarse:{level}"          # test 1: geometry precision

    if len(hits) > 1:
        gap = float(top.get("importance", 0)) - float(hits[1].get("importance", 0))
        if gap < min_gap:
            return None, f"ambiguous:gap={gap:.3f}"  # test 2: score separation

    return top, "ok"                                 # test 3 is spatial, later


PRECISION_RANK = {"house": 0, "building": 0, "office": 0, "amenity": 0,
                  "road": 1, "postcode": 2, "suburb": 3, "city": 4,
                  "state": 5, "country": 6}

The second test is the one people skip and the one that catches the most damage: a top result that barely beat the runner-up is a coin flip the geocoder made on your behalf.

Step-by-step solution #

1. Find out what the score actually measures #

Read the provider's definition before using the number, because the same field name means different things:

• String similarity — how closely the returned address text matches the query. Useful; comparable within one provider.
• Prominence or importance — how notable the matched feature is. Nominatim's importance is this, derived partly from Wikipedia linkage. It says nothing about your query.
• Composite quality — a blend of match completeness, reference-data quality and geometry precision, usually undocumented in detail.
• Categorical match codes — not a number at all: Exact, UpHierarchy, Ambiguous. These are the most honest, because they cannot be averaged.

The single most useful sentence in any geocoding API's documentation is the one defining this field. If it does not exist, treat the number as a ranking key and nothing more.

2. Separate precision from confidence, permanently #

They fail independently, so they need separate columns:

Only the top-left cell is safe to use unexamined. The top-right is the dangerous one, because it looks precise, and the bottom-left is the one that produces neat maps of wrong places.

3. Use the gap between candidates as the ambiguity signal #

Ask for three results instead of one. The absolute score is not comparable across queries; the difference between the top two is, because both were scored by the same function on the same query.

A large gap means the geocoder had a clear winner. A small gap means it picked one of several equals — and with limit=1 you would never have known. Place names are duplicated on a scale that makes this common: in the GeoNames gazetteer, 132,528 distinct place names occur in more than one country, San Antonio names 2,382 places across 25 countries, and Springfield names 68 places in the United States alone.

4. Calibrate the threshold on labelled data, once #

If you are going to use a numeric cut, earn it: take 200 addresses whose true locations you know — from a previous verified geocode, a survey, or a manual check — and plot the error distance against the score.

You will usually find the relationship is weak and steppy rather than smooth: the score separates "found the right kind of thing" from "did not", and within each precision level it carries almost no information about distance. That is a finding, not a failure — it tells you the threshold belongs on the precision field.

5. Validate spatially, because that is the test that does not lie #

Whatever the score says, the coordinate can be checked against something you already believe: the region the record claims to be in, the postcode polygon, the country boundary, a previous geocode of the same address.

The spatial test is independent of the provider's opinion, which is exactly what makes it worth more than the provider's opinion.

Code examples #

Example 1 — a quality record, not a quality number #

from dataclasses import dataclass


@dataclass
class MatchQuality:
    precision: str              # what kind of feature matched
    score: float | None         # the provider's number, unmodified
    score_gap: float | None     # top score minus runner-up
    candidates: int             # how many were returned
    bbox_span_km: float | None  # how big the matched feature is
    text_similarity: float      # our own comparison, provider-independent

    @property
    def verdict(self) -> str:
        if self.precision in ("country", "state", "region"):
            return "unusable"
        if self.score_gap is not None and self.score_gap < 0.05:
            return "ambiguous"
        if self.text_similarity < 0.6:
            return "suspect"
        if self.precision in ("house", "building", "office"):
            return "good"
        return "coarse"


def text_similarity(query: str, matched: str) -> float:
    """A provider-independent second opinion on the text match."""
    import difflib
    from parse_addresses import normalise          # your own normaliser
    return difflib.SequenceMatcher(
        None, normalise(query), normalise(matched)
    ).ratio()

Computing your own text similarity is worth the four lines: it is the one quality signal that means the same thing across every provider, which makes multi-provider tables comparable.

Example 2 — measuring whether the score predicts anything #

import statistics


def calibrate(labelled, geocoder):
    """labelled: [(query, true_lat, true_lon), ...] with verified truth."""
    buckets = {}
    for query, tlat, tlon in labelled:
        hits = geocoder.search(query)
        if not hits:
            buckets.setdefault("no match", []).append(None)
            continue
        top = hits[0]
        err = haversine_m((tlat, tlon), (float(top["lat"]), float(top["lon"])))
        band = round(float(top.get("importance", 0)) * 10) / 10
        buckets.setdefault(band, []).append(err)

    print(f"{'score':>6} {'n':>5} {'median err':>12} {'90th pct':>10}")
    for band in sorted(b for b in buckets if b != "no match"):
        errs = sorted(e for e in buckets[band] if e is not None)
        if not errs:
            continue
        print(f"{band:6.1f} {len(errs):5d} {statistics.median(errs):10,.0f} m "
              f"{errs[int(0.9 * len(errs)) - 1]:8,.0f} m")

Run this once per provider. If the median error is flat across score bands, the score is a ranking key, not a quality measure — and every threshold argument you were about to have is settled.

Example 3 — the ambiguity check that actually pays #

def ambiguity_report(queries, geocoder, gap_threshold=0.05):
    """How much of this file is the geocoder guessing at?"""
    guessed, clear, missing = [], 0, 0
    for q in queries:
        hits = geocoder.search(q, limit=3)
        if not hits:
            missing += 1
            continue
        if len(hits) == 1:
            clear += 1
            continue
        gap = float(hits[0].get("importance", 0)) - float(hits[1].get("importance", 0))
        if gap < gap_threshold:
            guessed.append((q, hits[0]["display_name"], hits[1]["display_name"], round(gap, 3)))
        else:
            clear += 1

    print(f"clear   {clear:5d}")
    print(f"guessed {len(guessed):5d}   <- these need a human or more context")
    print(f"missing {missing:5d}")
    for q, a, b, gap in guessed[:5]:
        print(f"  {q!r}\n    {gap}: {a[:60]}\n         vs {b[:60]}")
    return guessed

The output of this function is the most useful thing you can hand a domain expert: a short list of specific decisions the machine could not make, rather than a whole file to check.

Explanation #

Why scores are not comparable between providers #

Each provider computes its number from a different mixture of ingredients — string distance, feature prominence, reference-data confidence, population, query completeness — and normalises it into a different range with a different shape.

A 0.8 from one provider and a 0.8 from another have no relationship whatever. This matters in practice because multi-provider pipelines are normal: one geocoder for the domestic file, another for the international rows. Any threshold has to be per provider, and any comparison has to go through a signal you computed yourself.

Why prominence-based scores invert the ranking you want #

Nominatim's importance measures notability. That is the right ranking for a search box — someone typing "London" almost always means the big one — and the wrong ranking for a batch of addresses, where fame is uncorrelated with correctness.

The measured example is stark: the country of the United Kingdom scores 0.9389 and the specific building you asked for scores 0.5506. Sorting a file by that score puts the least useful answers at the top.

Why the score cannot know how far away the coordinate is #

The geocoder scores the match between two pieces of text. The distance error comes from the reference geometry — whether it stores that building as a point, interpolates it along a street segment, or has only the postal area.

Those two things are almost independent. A perfect text match against a street-interpolated reference is a confident answer that is 40 m out; a mediocre text match against an address-point reference is an uncertain answer that is exact. This is why precision has to be a separate field and why the calibration in Example 2 so often comes out flat.

Why "no match" is a quality signal, not a failure #

A geocoder that returns nothing has told you something true. One that returns a city centroid for the same query has told you something false in a format that looks identical to a success.

Providers differ in how willing they are to say nothing, and this is worth testing during selection. A higher no-match rate with honest reporting is generally cheaper to work with than a 100% hit rate padded with fallbacks — the first produces a work queue, the second produces a map that is quietly wrong.

Edge cases or notes #

• Ask for more than one candidate during development, even if production uses one. The gap is free information.
• Store the score, the precision and the candidate count. Any one of them alone is insufficient.
• Scores drift between provider versions. A threshold calibrated last year is not calibrated now.
• Empty queries and single-token queries produce high scores against countries and regions; filter them before sending.
• Structured queries usually score differently from free text on the same provider — calibrate the mode you use.
• A limit=1 request hides the ambiguity entirely and is the single most common cause of confident wrong answers.
• Match codes beat numbers. If a provider offers categorical codes, prefer them; they cannot be averaged into nonsense.
• Do not average scores across a file as a quality metric. Report the distribution by precision level instead.

Internal links #

• Geocoding explained: from an address string to a coordinate — the pipeline the score comes out of
• Rooftop, interpolated or centroid: geocoding precision levels — the field to filter on instead
• How to validate geocoding results before you trust them — the spatial tests
• Choosing a geocoder: coverage, licence and cost compared — comparing providers on more than the score
• Fixing a geocoder that matches the wrong town — the ambiguity failure in practice
• Address matching explained: why exact string equality fails — computing your own similarity
• How to geocode with Nominatim from Python without being blocked — where importance comes from
• Spatial data quality dimensions — quality as a set of measurable properties

FAQ #

What confidence threshold should I use? #

None, on its own. Filter on the precision level, flag results where the top two candidates are within about 0.05 of each other, and validate the coordinate against a boundary you trust.

Why does a country score higher than the exact address? #

Because some scores measure prominence rather than match quality. Nominatim's importance gave the United Kingdom 0.9389 and the building at 10 Downing Street 0.5506.

Can I compare confidence scores between two geocoders? #

No. Each provider computes and normalises the number differently. Compare a signal you compute yourself, such as string similarity between the query and the returned address.

What does a small gap between the top two results mean? #

That the geocoder picked one of several equally good candidates. With 132,528 place names occurring in more than one country, that is common — and with limit=1 it is invisible.

Does a high score mean the coordinate is accurate? #

No. The score is about the text match; the distance error comes from the reference geometry. Calibrate on labelled data and you will usually find the relationship is weak within a precision level.

Is a no-match better than a low-confidence match? #

Usually yes. A no-match produces a work queue; a low-confidence fallback produces a coordinate that looks like every other coordinate in the column.