My Model Scores 0.95 in Testing and Fails in the Field

Problem statement

A model with an excellent cross-validation score produces a map that domain experts reject. The score was not a lie — it answered a different question from the one deployment asks.

Measured on a real problem, the same model scored three ways:

random 5-fold CV, inside the sampled area      +0.555
true error over the whole unsampled map        +0.557
the same model applied to the adjacent half    -0.771

Random CV was accurate to 0.002 R² for the task it matched, and a different task on ground five kilometres away gave a negative R² — worse than predicting the training mean.

Quick answer

Work through five checks, in order of how often they are the cause:

1. is the deployment task the same as the CV task?
2. do the covariates in the new area resemble the training range?
3. did anything leak — coordinates, a proxy for the target, duplicated rows?
4. is the target distribution the same?
5. is the model extrapolating, and how does it behave when it does?

for i, name in enumerate(feature_names):
    train, new = X_train[:, i], X_new[:, i]
    lo, hi = np.percentile(train, [1, 99])
    outside = float(((new < lo) | (new > hi)).mean())
    print(f"  {name:12} {outside:6.1%} outside the training range")

A cross-validation score matching the interpolation task and diverging sharply for a new-region task. — The score is correct for the question it asked. The gap is between that question and deployment.

Step-by-step solution

1. Compare the CV task with the deployment task

The most common cause, and the easiest to miss because both are called "prediction".

If you validated by holding out random rows and you deploy to a new region, you measured interpolation and deployed extrapolation. The measured gap between those two on one dataset was 0.555 against −0.771.

Note that block CV does not close it: it promised +0.385 for the same transfer.

2. Check covariate shift

A model can only interpolate the feature space it saw. If the new area's elevations, slopes or spectral values lie outside the training range, every prediction there is extrapolation.

A feature with 30% of its new values outside the training 1st-to-99th percentile band is a strong signal, and tree models handle it particularly badly.

3. Look for leakage

Three kinds, in decreasing order of subtlety:

Spatial — random splits put near neighbours in both folds.
Target proxy — a covariate derived from the target, or measured at the same time by the same process.
Duplicated rows — repeated samples from one site, which appear in both folds under any splitter.

The tell for a proxy is a single feature dominating importance and a score that is too good. A soil model with R² = 0.98 from remote sensing alone is usually predicting the survey campaign rather than the soil.

4. Compare the target distributions

print(f"train: mean {y_train.mean():.3f}, sd {y_train.std():.3f}")
print(f"new:   mean {y_new.mean():.3f}, sd {y_new.std():.3f}")

If the new area's target has a different mean or spread, the model was fitted on a different problem. That is not fixable by tuning.

5. Understand how your model extrapolates

A random forest cannot predict outside the range of the training target — it averages training values in a leaf. In a new area it saturates at a plausible-looking constant, producing a flat map rather than an obviously wrong one.

Measured, adding coordinates as features made the transfer less wrong (−0.095 against −0.771) precisely because every new x value was outside the training range, so the model became nearly constant. A more "conservative" model, by accident.

Linear models do the opposite: they extrapolate confidently and diverge.

A random forest saturating at a constant outside its training range, against a linear model diverging. — Tree models fail flat; linear models fail loud. Neither is safe, and they look different on a map.

Code examples

Example 1 — a deployment audit

import numpy as np
from scipy.stats import ks_2samp


def deployment_audit(X_train, X_new, feature_names, y_train=None, y_new=None):
    """Everything that differs between training and deployment."""
    print(f"  {len(X_train):,} training rows, {len(X_new):,} deployment rows")

    problems = []
    for i, name in enumerate(feature_names):
        train, new = X_train[:, i], X_new[:, i]
        lo, hi = np.percentile(train, [1, 99])
        outside = float(((new < lo) | (new > hi)).mean())
        shift = float((new.mean() - train.mean()) / (train.std() + 1e-12))
        statistic, p_value = ks_2samp(train, new)

        flag = ""
        if outside > 0.10:
            flag = "  <- extrapolating"
            problems.append(f"{name}: {outside:.0%} outside the training range")
        elif abs(shift) > 1.0:
            flag = "  <- shifted"
            problems.append(f"{name}: mean shifted {shift:+.1f} sd")

        print(f"  {name:14} {outside:6.1%} outside  "
              f"{shift:+6.2f} sd shift  KS {statistic:.3f}{flag}")

    if y_train is not None and y_new is not None:
        print(f"  target: train mean {y_train.mean():.3f} "
              f"sd {y_train.std():.3f}; "
              f"new mean {y_new.mean():.3f} sd {y_new.std():.3f}")

    for p in problems:
        print(f"  ! {p}")
    return problems

The Kolmogorov-Smirnov statistic catches distribution changes that a mean shift misses — a feature with the same mean and a different shape.

Example 2 — checking for leakage

import numpy as np
from sklearn.metrics import r2_score


def leakage_checks(X, y, coords, feature_names, model_factory):
    """Three tests that catch the common leaks."""
    # 1. a single dominant feature is suspicious
    model = model_factory().fit(X, y)
    importance = getattr(model, "feature_importances_", None)
    if importance is not None:
        order = np.argsort(-importance)
        top = importance[order[0]]
        print(f"  top feature: {feature_names[order[0]]} ({top:.1%})")
        if top > 0.6:
            print("  ! one feature dominates — check it is not derived from "
                  "the target")

    # 2. duplicate or near-duplicate coordinates
    from scipy.spatial import cKDTree
    distances, _ = cKDTree(coords).query(coords, k=2)
    duplicates = int((distances[:, 1] < 1e-6).sum())
    if duplicates:
        print(f"  ! {duplicates:,} rows share a location with another row — "
              "these will land in different folds")

    # 3. how much better is random CV than a spatial split?
    from sklearn.cluster import KMeans
    from sklearn.model_selection import GroupKFold, KFold

    def score(splitter):
        predicted = np.empty(len(y))
        for train, test in splitter:
            predicted[test] = model_factory().fit(X[train], y[train]) \
                                             .predict(X[test])
        return r2_score(y, predicted)

    random = score(KFold(5, shuffle=True, random_state=0).split(X))
    blocks = KMeans(10, random_state=0, n_init=10).fit_predict(coords)
    spatial = score(GroupKFold(5).split(X, y, blocks))

    print(f"  random CV {random:+.3f}, block CV {spatial:+.3f}, "
          f"gap {random - spatial:+.3f}")
    if random - spatial > 0.3:
        print("  ! large gap — the model relies heavily on spatial proximity")
    return {"random": random, "block": spatial}

A large random-block gap is not proof of a problem — it was 0.34 on a model that was genuinely accurate for its task. It is a signal that the model leans on location, which matters if you will deploy elsewhere and does not if you will not.

Example 3 — an applicability mask to ship with the map

import numpy as np
from scipy.spatial import cKDTree
from sklearn.preprocessing import StandardScaler


def applicability_mask(X_train, X_new, quantile=0.95, k=5):
    """Flag predictions whose covariates are unlike anything in training."""
    scaler = StandardScaler().fit(X_train)
    train_scaled = scaler.transform(X_train)
    new_scaled = scaler.transform(X_new)

    tree = cKDTree(train_scaled)
    train_distances, _ = tree.query(train_scaled, k=k + 1)
    threshold = float(np.quantile(train_distances[:, 1:].mean(axis=1), quantile))

    new_distances, _ = tree.query(new_scaled, k=k)
    new_mean = new_distances.mean(axis=1)
    inside = new_mean <= threshold

    print(f"  threshold: {quantile:.0%} of training points are within "
          f"{threshold:.3f} (scaled feature-space distance)")
    print(f"  {inside.mean():.1%} of deployment locations are inside the "
          "area of applicability")
    return inside, new_mean

Shipping this mask with the prediction is the practical answer to "we cannot validate transfer". It does not tell you the error outside the mask; it tells you where the model is guessing, which is what a map user needs.

Explanation

Why the score can be exactly right and still useless

Cross-validation estimates generalisation to data drawn from the same distribution as the training data. That is a precise statement and a narrow one.

Random CV on a dense random sample estimates interpolation within the sampled extent, and it does so very well — +0.555 against a true +0.557 in the measurement here.

Deployment to a new region asks about a different distribution. No resampling of the training data can estimate it, because the training data contains no information about the difference.

Why tree models fail quietly

A random forest predicts the mean of the training targets in a leaf. It therefore cannot produce a value outside the training target range, and for a feature value beyond the training range it reuses the most extreme split it learned.

Predictions in a new area consequently converge on a constant. The map looks plausible — smooth, in range, no obvious artefacts — and carries no information.

That is worse than an obviously wrong map, which at least gets noticed. It is why an applicability mask matters more for tree models than for linear ones.

Why coordinates make transfer worse and the score better

Adding raw x and y let the model carve the study area into regions with different local means. That is genuinely useful for interpolation — the measured in-area R² rose from 0.559 to 0.721.

It is a lookup table for one place. Applied elsewhere, the splits are meaningless.

The measured transfer results are instructive: without coordinates, −0.771; with them, −0.095. The version with coordinates scored better because it had been broken into near-constancy, not because it transferred.

Why domain experts spot it first

A model's failure in a new area is usually a pattern failure: the map is smooth where it should be varied, or puts a class where everyone knows it does not occur.

Metrics do not capture pattern. An expert looking at the map sees it immediately, and the fastest route to catching this class of failure is to show a domain expert the map before the metrics.

Four covariates checked against the training range, with elevation 31 percent outside and flagged as extrapolating. — Run this before deploying anywhere new. It is four lines and it is the check that matters.

Edge cases or notes

A good score for the wrong task is still a good score. State the task.
Check covariate range overlap before deploying anywhere new.
Tree models saturate outside their range; linear models diverge.
Coordinates as features destroy transferability while improving in-area accuracy.
Duplicated sites leak under any splitter.
A dominant feature may be a target proxy.
A negative R² means worse than the mean — report it.
Show the map to a domain expert before trusting the metric.

Internal links

Spatial leakage explained: why random cross-validation lies — the measurements behind this
How to do spatial cross-validation in Python — choosing a scheme
Extrapolation in space explained: the area of applicability — the mask to ship
Spatial machine learning explained — why location breaks the usual rules
Feature importance says coordinates are the best predictor — the coordinate trap
How to evaluate a spatial model honestly — what to report
My prediction raster is striped, blocky or full of NoData — the visual symptoms
Spatial autocorrelation explained — the structure underneath

FAQ

Why does my model score well and fail in practice?

Almost always because the validation task and the deployment task differ. Random CV measures interpolation; deploying to a new region is extrapolation, and the measured gap was 0.555 against −0.771.

Does spatial cross-validation fix it?

Not for a new region. Block CV promised +0.385 where a real transfer test gave −0.771.

How do I know if I am extrapolating?

Compare each covariate's deployment values against the training 1st-to-99th percentile band. More than about 10% outside means extrapolation.

Why does my prediction map look flat in the new area?

A tree model cannot predict outside its training target range, so beyond the training feature space it saturates at a constant.

Should I include coordinates as features?

For interpolation inside the sampled area, they help — 0.559 to 0.721 here. For anything that must transfer, no.

What is a target proxy?

A covariate derived from, or measured alongside, the target. It produces implausibly high scores and does not exist at prediction time.

What should I ship with a prediction map?

An area-of-applicability mask showing where the covariates resemble the training data, and the validation scheme with the score.

My Model Scores 0.95 in Testing and Fails in the Field #

Problem statement #

Quick answer #

Step-by-step solution #

1. Compare the CV task with the deployment task #

2. Check covariate shift #

3. Look for leakage #

4. Compare the target distributions #

5. Understand how your model extrapolates #

Code examples #

Example 1 — a deployment audit #

Example 2 — checking for leakage #

Example 3 — an applicability mask to ship with the map #

Explanation #

Why the score can be exactly right and still useless #

Why tree models fail quietly #

Why coordinates make transfer worse and the score better #

Why domain experts spot it first #

Edge cases or notes #

Internal links #

FAQ #

Why does my model score well and fail in practice? #

Does spatial cross-validation fix it? #

How do I know if I am extrapolating? #

Why does my prediction map look flat in the new area? #

Should I include coordinates as features? #

What is a target proxy? #

What should I ship with a prediction map? #