Choosing classification metrics: precision, recall, F1, thresholds and calibration

What it is

The scikit-learn metrics guide describes precision as the classifier's ability not to label a negative sample as positive, recall as its ability to find all positive samples, and the F-measure as a weighted harmonic mean of the two, with F1 weighting both equally. These are computed from hard predictions at one decision threshold. Threshold-free views come from the precision-recall curve and average precision, or the ROC curve, which score how well the model ranks positives above negatives. Calibration is a separate property: the calibration guide explains that a well-calibrated classifier's predicted probabilities match observed frequencies, shows this with calibration curves (reliability diagrams), and notes that strictly proper scoring rules such as the Brier score and log loss assess calibration and discriminative power together, so they are of limited use for judging calibration alone; CalibratedClassifierCV post-processes an uncalibrated model.

Why it matters

A metric is a decision rule in disguise. A fraud filter that blocks transactions needs high precision; a screening step followed by human review needs high recall; a system that reports "probability 0.8" to a downstream rule needs calibration, not ranking. Optimising the wrong one produces a model that scores well and fails its purpose.

How to apply

• Write down, before modelling, what one false positive and one false negative cost or mean; that fixes the precision-recall trade-off and often the threshold.
• Report precision, recall and the confusion matrix at the chosen threshold, plus a threshold-free score for comparing models; the threshold is a deployment parameter, tuned on validation data, not part of the model.
• If the output is a probability that anyone will read as one, plot the calibration curve on held-out data and report the Brier score or log loss next to it, not instead of it.
• Report metrics per class and per relevant segment; a single number averaged over classes hides a class the model never predicts.
• Put an interval around the metric; on small test sets the difference between two models is often inside it.

Pitfalls

Accuracy on a 99:1 dataset rewards predicting the majority class. F1 ignores true negatives entirely, which is right for rare-positive problems and wrong for balanced ones. The calibration guide shows that bagged ensembles such as random forests push predicted probabilities away from 0 and 1, so a good ranking does not imply calibrated scores.