Overfitting and regularisation in outline: bias, variance and the penalty knob

What it is

The scikit-learn guide on validation curves decomposes an estimator's generalisation error into bias, variance and noise: bias is the average error across different training sets, variance is how sensitive the fitted model is to which training set it saw, noise belongs to the data. A model with too little capacity has high bias (underfits); one with too much capacity fits the training rows including their noise and has high variance (overfits). The guide's diagnostic is the pair of curves: a training score much higher than the validation score indicates overfitting, both low indicates underfitting, and a learning curve over increasing training size shows whether more data would help. Regularisation is the standard counter-measure; the linear-models guide describes ridge regression as addressing problems of ordinary least squares by imposing a penalty on the size of the coefficients, with a parameter alpha controlling the amount of shrinkage, and lasso as the L1 variant that drives some coefficients to exactly zero.

Why it matters

Overfitting is the default failure of any flexible model on a finite dataset, and it is invisible on the training score. The regularisation strength is the hyperparameter that directly moves a model along the bias-variance trade-off, which makes it the first one to tune.

How to apply

• Always look at training and validation scores side by side; the gap is the diagnostic, not either number alone.
• Tune the regularisation strength on the validation split or by cross-validation over a logarithmic grid; the right value depends on the data, not on defaults.
• Scale features before applying a coefficient penalty, since the penalty treats all coefficients alike: the same information expressed in large units needs only a small coefficient that the penalty barely touches, while in small units it needs a large, heavily penalised one.
• For trees and ensembles, the equivalent knobs are depth, minimum samples per leaf, number of trees and learning rate; for neural networks, weight decay, dropout and early stopping on validation loss.
• Prefer more or cleaner data over a cleverer model when the learning curve is still rising.

Pitfalls

Selecting the regularisation strength on the test set converts the test set into a validation set. Very strong regularisation underfits quietly and looks like "the data has no signal". Early stopping needs its own validation split, separate from the one used to report results.