Early stopping: patience, validation and restoring the best weights

What early stopping selects

Early stopping ends iterative training when a monitored quantity stops improving according to a specified rule. It can reduce computation and limit overfitting, but it does not guarantee either better generalization or the globally optimal stopping point. The number of training epochs becomes a model-selection decision. A validation signal therefore needs data separated from the examples used to update weights. [1][2]

Define the stopping rule

Specify the monitored metric, its direction, the minimum improvement and the patience. For validation loss, a smaller value is normally better. In Keras, min_delta describes an absolute improvement requirement, while patience counts epochs without qualifying improvement. A warm-up period can delay monitoring. [1]

Stopping and checkpoint selection are different choices. Setting restore_best_weights=True restores the weights associated with the best monitored value under the callback's comparison rule. With it disabled, the model retains the last trained weights. State these settings explicitly instead of assuming a framework restores automatically. [1]

Keep validation separate from final testing

Split data before learning transformations, and fit those transformations on the training partition. Use validation data to choose patience, the monitored metric, learning-rate settings and other hyperparameters. Do not inspect test loss every epoch or tune patience after seeing test performance: both practices let test information select the model. [2]

For cross-validation, keep the early-stopping data inside each training fold; the fold used for evaluation must remain separate. Record the split, seed, metric history, best epoch and stopping epoch. This workflow is a practical consequence of preserving independent evaluation.

Illustrative epoch sequence

Consider validation losses of 0.42, 0.39, 0.40, 0.395, 0.398 for epochs 1–5. With minimization, min_delta=0 and patience=3, epoch 2 is the best; the next three epochs do not improve it. Training stops after epoch 5, and restoring the best weights returns to epoch 2. This constructed sequence explains the rule; it is not an experimental result. [1]

Limits and reporting

Noisy or unrepresentative validation data can stop training too early or select a misleading checkpoint. A metric plateau may also precede improvement after a planned learning-rate change. Choose the monitoring policy before final testing and align its metric with the task. Repeated validation tuning can itself overfit validation data. Evaluate the selected model on untouched test data and report both epochs and the stopping configuration. No local training experiment was performed.

Sources

[1] TensorFlow API: tf.keras.callbacks.EarlyStopping

[2] scikit-learn User Guide: Common Pitfalls and Recommended Practices