Temperature scaling: calibrating classifier confidence

Definition

Temperature scaling is a post-training method for adjusting a classifier’s probability estimates. It divides all logits for an example by the same positive scalar before applying softmax. The objective is better agreement between predicted confidence and observed outcomes; it does not retrain the classifier’s feature representation. [1]

How it works

For logits z and temperature T > 0, the calibrated probabilities are softmax(z/T). T = 1 leaves them unchanged; T > 1 softens the distribution, while 0 < T < 1 sharpens it. Learn T by minimizing negative log-likelihood on a separate calibration sample.

A positive common divisor preserves the ordering of classes within each example. Thus argmax predictions, and accuracy computed from those predictions, remain unchanged. Decisions using a fixed probability threshold can nevertheless change. [1]

Practical workflow

1. Separate data for model fitting, calibration and final evaluation, respecting time or entity boundaries when relevant.
2. Freeze the trained classifier and collect its calibration logits and corresponding labels.
3. Optimize one positive temperature; record the dataset, objective and selected value alongside the model version.
4. On untouched evaluation data, compare reliability curves with bin counts, log loss, Brier score and argmax accuracy.

Choose acceptance criteria before inspecting the evaluation results. A downstream rejection or routing threshold should be evaluated with the calibrated probabilities actually used in production.

Illustrative example

Illustrative calculation, not a measured calibration result: two logits are 3 and 1. Their first-class probability is approximately 0.881. With a hypothetical T = 2, the logits become 1.5 and 0.5, producing approximately 0.731. The first class still wins.

A workflow that automatically routes cases above 0.80 would route the original prediction but withhold the scaled one. Therefore unchanged classification accuracy does not imply unchanged operational behaviour. This example demonstrates the transformation, not evidence that T = 2 is appropriate for a particular dataset.

Limitations and evaluation

A single temperature cannot introduce class-specific offsets or correct every pattern of miscalibration. Calibration data must represent the intended use; the fitted value provides no automatic guarantee for new distributions or small subgroups.

Reliability diagrams depend on binning and sample counts. Brier score and log loss evaluate probability quality more broadly, combining calibration and discrimination, so improvement in either is not proof of improved calibration alone. [2] Report the calibration sample size and retain a comparison with the unscaled model. No local model training or benchmark is claimed here.

Sources

[1] On Calibration of Modern Neural Networks [2] scikit-learn: Probability calibration