Handling class imbalance: metrics first, then weights, thresholds and resampling inside the pipeline

What it is

Imbalance means the classes occur at very different rates: fraud among transactions, failures among machines, one diagnosis among visits. A classifier trained on such data with a plain loss and evaluated with accuracy can predict the majority class for every row and look excellent. The scikit-learn metrics guide describes balanced accuracy as the macro-average of per-class recall, which avoids inflated estimates on imbalanced datasets. Estimators such as LogisticRegression accept a class_weight parameter, with balanced setting weights inversely proportional to class frequencies, so that errors on the rare class cost more during fitting. The imbalanced-learn library adds resampling (random over- and under-sampling, SMOTE, which synthesises minority examples); its pitfalls page warns that resampling the whole dataset before splitting causes data leakage and evaluates the model on an artificially balanced set that does not resemble real use.

Why it matters

The rare class is usually the one that matters, and it is the one a naive setup makes invisible. A "99 percent accuracy" figure on imbalanced data may describe the class ratio rather than the model.

How to apply

• Split with stratification so every fold contains the minority class; report precision, recall and F1 for the minority class, the confusion matrix, and a threshold-free score such as average precision.
• Try class weights before resampling; they change the loss without changing the data and keep the evaluation set natural.
• Tune the decision threshold on validation data for the cost trade-off instead of accepting 0.5.
• If resampling, put the sampler inside the pipeline that is cross-validated, so it runs on the training fold only and the validation fold keeps its natural ratio.
• Check calibration afterwards: weighting and resampling shift predicted probabilities away from the true base rate, which matters if the score is used as a probability.

Pitfalls

Resampling before the split leaks synthetic copies of validation rows into training. On a balanced dataset balanced accuracy equals accuracy, as the metrics guide states, so it adds nothing there. A minority class with a handful of examples cannot be learned by any reweighting; more data or a different problem framing is needed.

Threshold before correction

For a model that outputs probabilities, imbalance does not by itself degrade the ranking of positives over negatives; it makes the default cut-off of 0.5 wrong. Tune the decision threshold on validation data against the cost trade-off first, with the model fitted on the natural class ratio. Class weights and resampling change the fitted probabilities, so a model corrected that way needs a separate calibration step before its scores can be read as probabilities. Reach for weights or resampling only when the ranking itself is poor and more minority data or better features are unavailable, and report calibration on held-out data whenever they are used.