⚙️ How-To Spatial machine learning Spatial Machine Learning · step 14 of 14 Updated

How to Handle Class Imbalance in a Spatial Classifier

Problem statement

Land cover classes are never evenly distributed. A class covering 2% of the landscape gets 2% of a random sample, which is rarely enough to learn — and a classifier that ignores it entirely still scores 98% accuracy.

Spatial data adds a complication that generic imbalance advice misses: rare classes are clustered. A rare wetland class is not 2% of every neighbourhood; it is 100% of a few places and 0% everywhere else.

That means a random split can put an entire class in one fold, and resampling techniques that assume independent rows create near-duplicates that leak straight across folds.

Quick answer

Stratify the sample, weight the classes, and never report accuracy alone:

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import balanced_accuracy_score, classification_report

model = RandomForestClassifier(
    n_estimators=200, min_samples_leaf=2,
    class_weight="balanced_subsample", n_jobs=-1, random_state=0)

print(classification_report(y_true, y_pred, digits=3))
print(f"balanced accuracy {balanced_accuracy_score(y_true, y_pred):.3f}")

Balanced accuracy is the mean per-class recall, so a class the model never predicts contributes zero rather than being hidden.

A classifier ignoring a 2 percent class and still scoring 98 percent accuracy, against balanced accuracy which drops sharply.
Overall accuracy is dominated by the majority class. Balanced accuracy is not.

Step-by-step solution

1. Stratify the training sample 

per_class = 300
for label in classes:
    idx = np.nonzero(labels == label)[0]
    picked.append(rng.choice(idx, min(per_class, len(idx)), replace=False))

Equal counts per class give the model enough examples of each to learn a boundary. It also changes the class prior, so predicted probabilities no longer reflect landscape frequencies — a trade to make deliberately.

Set a minimum: a class with fewer than about 30 samples cannot be learned reliably, and including it produces a class the model predicts almost at random.

2. Keep spatial groups intact

A rare class often comes from a handful of polygons. Random splitting puts samples from one polygon in both folds, and the model memorises the polygon rather than the class.

Group by polygon, plot or survey unit, and use GroupKFold or StratifiedGroupKFold. Where a class exists in only two polygons, no split can validate it honestly — say so rather than reporting a number.

3. Prefer class weights to resampling

class_weight="balanced" reweights the loss so each class contributes equally, without duplicating rows.

Oversampling duplicates rows, which under any spatial validation puts identical rows in different folds. Synthetic oversampling such as SMOTE interpolates between neighbours in feature space, creating rows that are spatially meaningless and leak in the same way.

For spatial data, weighting is almost always the better tool.

4. Report per-class metrics 

              precision  recall  f1-score  support
vegetation        0.94    0.97      0.95     4200
bare soil         0.71    0.58      0.64      340
water             0.89    0.92      0.90      210
wetland           0.42    0.19      0.27       48

Recall is what matters for a rare class: 0.19 means four in five wetland cells were missed. That is invisible in an overall accuracy of 0.93.

5. Set the threshold from the cost, not from the default

For a two-class problem, the 0.5 default is arbitrary. If missing the rare class is expensive and a false positive is cheap, lower it — and report the threshold with the results.

A rare class occupying a few contiguous patches rather than being scattered, so a random split puts the same patch in both folds.
Rare classes are clustered, so random splits validate memorisation of a patch rather than knowledge of a class.

Code examples

Example 1 — stratified, grouped sampling 

import numpy as np


def stratified_grouped_sample(labels, groups, per_class=300, min_class=30,
                              min_groups=3, seed=0):
    """Equal-ish rows per class, drawn from as many groups as possible."""
    rng = np.random.default_rng(seed)
    picked, notes = [], []

    for label in np.unique(labels):
        idx = np.nonzero(labels == label)[0]
        available_groups = np.unique(groups[idx])

        if len(idx) < min_class:
            notes.append(f"class {label}: only {len(idx)} samples — dropped")
            continue
        if len(available_groups) < min_groups:
            notes.append(f"class {label}: only {len(available_groups)} "
                         "spatial groups — cannot be validated honestly")

        per_group = max(1, per_class // len(available_groups))
        chosen = []
        for group in available_groups:
            in_group = idx[groups[idx] == group]
            chosen.append(rng.choice(in_group,
                                     min(per_group, len(in_group)),
                                     replace=False))
        chosen = np.concatenate(chosen)
        if len(chosen) > per_class:
            chosen = rng.choice(chosen, per_class, replace=False)
        picked.append(chosen)

        print(f"  class {label}: {len(idx):7,} available in "
              f"{len(available_groups):3d} groups -> {len(chosen):5,} taken")

    for note in notes:
        print(f"  ! {note}")
    return np.concatenate(picked)

Spreading the sample across groups is what stops a class being represented by one polygon. A class drawn entirely from one wetland teaches the model that wetland looks like that wetland.

Example 2 — weighting rather than resampling 

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import balanced_accuracy_score, classification_report
from sklearn.model_selection import StratifiedGroupKFold


def fit_balanced(X, y, groups, class_names=None, n_splits=5, seed=0):
    """Class-weighted forest, validated with groups kept together."""
    classes, counts = np.unique(y, return_counts=True)
    print(f"  {len(classes)} classes, "
          f"imbalance ratio {counts.max() / counts.min():.1f}:1")
    for label, count in zip(classes, counts):
        name = class_names.get(label, label) if class_names else label
        print(f"    {name:18} {count:7,} ({count / len(y):6.2%})")

    predicted = np.empty(len(y), dtype=y.dtype)
    splitter = StratifiedGroupKFold(n_splits=n_splits, shuffle=True,
                                    random_state=seed)
    for train, test in splitter.split(X, y, groups):
        model = RandomForestClassifier(
            n_estimators=200, min_samples_leaf=2,
            class_weight="balanced_subsample", n_jobs=-1, random_state=seed)
        predicted[test] = model.fit(X[train], y[train]).predict(X[test])

    print(classification_report(y, predicted, digits=3,
                                target_names=[str(class_names.get(c, c))
                                              for c in classes]
                                if class_names else None))
    print(f"  balanced accuracy {balanced_accuracy_score(y, predicted):.3f}")
    print(f"  overall accuracy  {(predicted == y).mean():.3f}")
    return predicted

Printing both accuracies side by side makes the imbalance visible in one line. A large gap between them means the model is doing well on the common classes and poorly on the rare ones.

class_weight="balanced_subsample" recomputes the weights for each tree's bootstrap sample, which suits a forest better than a single global weighting.

Example 3 — a confusion matrix that shows where the errors go 

import numpy as np
from sklearn.metrics import confusion_matrix


def confusion_report(y_true, y_pred, class_names=None, normalise="true"):
    """Row-normalised confusion, so recall is readable directly."""
    labels = np.unique(np.concatenate([y_true, y_pred]))
    matrix = confusion_matrix(y_true, y_pred, labels=labels,
                              normalize=normalise)
    names = [str(class_names.get(l, l)) if class_names else str(l)
             for l in labels]
    width = max(len(n) for n in names) + 1

    print(" " * (width + 2) + " ".join(f"{n[:6]:>7}" for n in names))
    for i, name in enumerate(names):
        row = " ".join(f"{matrix[i, j]:7.2f}" for j in range(len(labels)))
        print(f"  {name:{width}} {row}")

    for i, name in enumerate(names):
        confused_with = np.argsort(-matrix[i])
        worst = [j for j in confused_with if j != i][0]
        if matrix[i, worst] > 0.15:
            print(f"  ! {name} is confused with {names[worst]} "
                  f"{matrix[i, worst]:.0%} of the time")
    return matrix

Row normalisation makes each row's diagonal the recall for that class, and the off-diagonals show exactly which class the errors go to. That is far more actionable than a single score: "wetland is being called water 40% of the time" points at a feature that could separate them.

Explanation

Why rare classes are clustered, and why that matters

A rare land cover class is not scattered uniformly at 2% density. It occupies a few patches where it is the dominant class.

That has two consequences generic imbalance advice does not address.

A random split puts samples from one patch in both training and test folds, so the model memorises the patch and the validation rewards it. The reported recall for the rare class is then optimistic by a large margin.

And a class present in only two or three patches cannot be validated at all: any split either trains on all of them or tests on a patch the model has never seen anything like.

Why weighting beats resampling here

Oversampling duplicates rows. Under GroupKFold the duplicates stay together, but under any random element they can separate, and a duplicated row in both folds is perfect leakage.

Synthetic methods such as SMOTE interpolate between neighbours in feature space to create new rows. Those rows have no location, so they cannot be assigned to a spatial group, and they are typically dropped from spatial splitting or assigned arbitrarily.

Class weighting changes the loss without changing the rows, so the spatial structure — and therefore the validation — stays intact.

Why accuracy is the wrong headline

With one class at 90% of the landscape, a model that always predicts it scores 0.90. That is a strong-sounding number for a model that has learned nothing.

Balanced accuracy — the mean recall across classes — gives that model 1/n, where n is the number of classes. For five classes that is 0.20, which is a fairer description.

Per-class recall is more informative still, because it says which classes are being missed, and that usually points at a specific confusion worth fixing.

Why stratification changes what the probabilities mean

A classifier learns the class frequencies of its training data. Training on 300 samples of each of five classes teaches it that all five are equally likely.

The decision boundaries usually improve, because the rare classes get enough examples. The predicted probabilities become conditional on the sampling design rather than on the landscape.

Where the probabilities matter — a suitability surface, a risk map — correct them back to the landscape priors, or weight rather than resample so the priors are preserved.

Class weighting reweighting the loss against oversampling creating duplicate and synthetic rows that leak across spatial folds.
Synthetic rows have no location, so they cannot be grouped — and grouping is the whole defence.

Edge cases or notes

  • Rare classes are clustered. Group by patch and use StratifiedGroupKFold.
  • A class in fewer than about three groups cannot be validated honestly.
  • Prefer class_weight to resampling — duplicates and synthetic rows leak.
  • Report balanced accuracy and per-class recall, never accuracy alone.
  • Stratifying changes the prior, so probabilities need correcting.
  • Set a minimum class size, around 30 samples.
  • Row-normalise the confusion matrix so the diagonal is recall.
  • Report the decision threshold if you moved it from the default.

FAQ

How do I handle class imbalance in a land cover classifier?

Stratify the sample so each class has enough rows, use class_weight="balanced" rather than resampling, keep spatial groups intact when splitting, and report per-class recall.

Why not use SMOTE or oversampling?

Both create rows that leak across spatial folds — duplicates directly, synthetic rows because they have no location and cannot be grouped.

Why is my accuracy high but the map wrong?

Overall accuracy is dominated by the majority class. A model ignoring a 2% class still scores 98%. Report balanced accuracy and per-class recall.

How many samples does a rare class need?

At least about 30, and from at least three separate spatial groups. Fewer than that and it cannot be validated honestly.

Does stratified sampling bias my model?

It changes the class prior, so predicted probabilities no longer reflect landscape frequencies. Decision boundaries usually improve. Correct the priors if the probabilities matter.

What metric should I report?

Per-class precision and recall, plus balanced accuracy. A row-normalised confusion matrix shows which classes are being confused.

Should I move the decision threshold?

If the costs of the two error types differ, yes — and report the threshold with the results.