Topic: deep-learning
-
Gradient accumulation: effective batches without losing the correct normalization
Accumulate microbatch gradients to reduce activation memory, while preserving loss weighting, optimizer boundaries and clipping order; a larger effective batch is not always equivalent to one physical batch.
-
Knowledge distillation: transferring useful predictions from teacher to student
Train a student against both labels and a teacher’s probability distribution, with explicit temperature, loss weighting and independent evaluation. Compression benefits are conditional on teacher quality, student capacity and deployment behavior.
-
LoRA: low-rank adaptation and what its memory savings actually cover
LoRA learns a low-rank update while keeping base weights frozen. It reduces trainable parameters and their optimizer state, but the base model, activations and deployment compatibility still matter.
-
Mixed precision training: autocasting, loss scaling and numerical limits
Mixed precision assigns suitable floating-point formats to individual operations. Autocasting and loss scaling address different problems, and performance gains require both numerical checks and measurements on the intended hardware.
-
AdamW: why decoupled weight decay differs from an L2 penalty
AdamW shrinks parameters separately from the adaptive gradient update. This prevents the decay term entering Adam’s moment estimates, but learning rate, update count and parameter-group choices still affect regularization.
-
Deep ensembles: predictive uncertainty and shared blind spots
Combine independently trained networks, distinguish within-model variance from disagreement and evaluate probability quality and operational cost under realistic shifts.
Machine-readable: JSON