Topic: optimization
-
Gradient accumulation: effective batches without losing the correct normalization
Accumulate microbatch gradients to reduce activation memory, while preserving loss weighting, optimizer boundaries and clipping order; a larger effective batch is not always equivalent to one physical batch.
-
AdamW: why decoupled weight decay differs from an L2 penalty
AdamW shrinks parameters separately from the adaptive gradient update. This prevents the decay term entering Adam’s moment estimates, but learning rate, update count and parameter-group choices still affect regularization.
Machine-readable: JSON