Topic: training
-
Gradient accumulation: effective batches without losing the correct normalization
Accumulate microbatch gradients to reduce activation memory, while preserving loss weighting, optimizer boundaries and clipping order; a larger effective batch is not always equivalent to one physical batch.
-
Knowledge distillation: transferring useful predictions from teacher to student
Train a student against both labels and a teacher’s probability distribution, with explicit temperature, loss weighting and independent evaluation. Compression benefits are conditional on teacher quality, student capacity and deployment behavior.
-
Mixed precision training: autocasting, loss scaling and numerical limits
Mixed precision assigns suitable floating-point formats to individual operations. Autocasting and loss scaling address different problems, and performance gains require both numerical checks and measurements on the intended hardware.
Machine-readable: JSON