Knowledge distillation: transferring useful predictions from teacher to student

What it is

Knowledge distillation uses a teacher model to guide a student. The student is often smaller, but the defining feature is the extra training signal rather than model size. In classical classification distillation, the teacher supplies a distribution over classes instead of only one winning label, exposing relationships among alternatives. [1]

How it works

Divide teacher and student logits by a temperature T before softmax. A higher temperature softens the distributions. A common objective is L = (1-alpha)*CE(y, p_s) + alpha*T^2*KL(p_t(T) || p_s(T)). Here p_s in the label term uses temperature 1; the distillation term compares both models at the same T. Alpha controls their relative weight. Multiplication by T squared compensates the temperature-related gradient scaling approximately; it is not a guarantee that every temperature behaves equally. [1][2]

Practical workflow

Establish separate training, validation and test data. A teacher must not introduce test-set information into student training. Freeze the teacher, use evaluation mode for ordinary offline distillation, and avoid storing its gradients. Feed teacher and student corresponding inputs with compatible class ordering. [2]

Choose a student baseline trained on labels alone, then tune temperature and loss weights on validation data. Implement KL with the library’s required probability or log-probability convention and a documented reduction over classes and examples. Keep the final test set untouched until selection is complete. Record teacher version, preprocessing, transfer data and student architecture.

Illustrative example

An illustrative support-ticket teacher assigns shipping 0.70, refund 0.25 and account 0.05. A hard shipping label hides the distinction between the two alternatives; the distribution indicates that refund is the closer competing interpretation. A student can learn that structure while still learning the true label. These chosen probabilities illustrate the idea and do not report performance or a particular temperature transformation.

Limits and verification

A teacher can transmit errors and biases. A very small student may lack capacity to reproduce useful distinctions, and domain shift can make confident teacher outputs misleading. Matching distributions is therefore not itself proof of better task performance. [1][2] Different label spaces or token vocabularies require an explicit alignment method; the simple classwise objective does not automatically apply.

Compare the distilled student, the label-only student and the teacher on independent task metrics, relevant subgroups, latency and memory. Evaluate deployed preprocessing and output behavior, not just the training loss. No local distillation experiment or compression benchmark was performed for this article.

Sources

[1] Hinton, Vinyals and Dean: Distilling the Knowledge in a Neural Network [2] PyTorch: Knowledge Distillation Tutorial