AdamW: why decoupled weight decay differs from an L2 penalty

What it is

AdamW combines Adam’s adaptive optimization with decoupled weight decay. It maintains gradient moment estimates for the optimization step while applying a separate shrinkage to selected parameters. This distinction matters because adding an L2 penalty to the loss is generally not equivalent to weight decay under an adaptive optimizer. [1][2]

How it works

Adding a penalty of lambda/2 times the squared parameter norm contributes lambda times the parameter to the gradient. With ordinary SGD without momentum, that term can be written as proportional parameter shrinkage, using matching coefficient conventions. Under Adam, however, the added term enters both moment estimates and their coordinate-specific scaling. AdamW keeps the decay out of those estimates. [1]

A simplified AdamW step is theta <- (1 - eta*lambda)*theta - eta*u, where eta is the learning rate, lambda the decay coefficient and u the bias-corrected adaptive gradient term. In PyTorch, decay still depends on the learning rate. Decoupling does not mean that shrinkage is independent of the learning-rate schedule. [2]

Practical workflow

Define explicit parameter groups and record which weights receive decay. Excluding biases or normalization parameters is a modeling choice to validate, not a universal law. Avoid adding an L2 loss term unintentionally on top of AdamW’s decay. Keep data splits and evaluation criteria fixed when comparing alternatives.

Tune learning rate and decay with the training duration in mind. Changing gradient accumulation changes the number of optimizer updates for a fixed number of examples, which also changes how often decay is applied. Store optimizer state and the schedule when resuming training; weights alone do not restore Adam’s history. [2]

Illustrative example

For a parameter currently equal to 2, let eta be 0.01 and lambda be 0.1. The decay contribution subtracts 0.002, leaving 1.998 before the adaptive gradient contribution. Halving eta halves this shrinkage for that step. These chosen values illustrate the equation; they are neither recommended hyperparameters nor an experimental finding.

Limits and verification

AdamW is not a guarantee of better generalization for every model or dataset. Strong decay can underfit, and zero decay remains a legitimate comparison. The original paper’s results establish behavior in its studied settings, not a universal ranking. [1]

Inspect the actual optimizer configuration, group membership, update counts and validation curves. Compare matched training budgets and report both predictive quality and stability. The checks proposed here have not been executed for this article; no local benchmark is claimed.

Sources

[1] Loshchilov and Hutter: Decoupled Weight Decay Regularization [2] PyTorch: AdamW