Training, validation and test sets: what each split is for and how to cut it

What it is

The scikit-learn user guide describes the standard arrangement: a model that is evaluated on the rows it was fitted on scores well and says nothing about unseen data, so part of the data is held out as a test set. Because hyperparameters are tuned by looking at scores, tuning against the test set lets knowledge of it leak into the model; the guide therefore introduces a third part, the validation set, on which choices are made, and states that a test set should still be held out for final evaluation even when cross-validation replaces the validation set. In k-fold cross-validation the training data is split into k parts and each part serves once as validation, so all training rows contribute to the model-selection estimate.

Why it matters

The numbers reported from the test set are the only ones that estimate production behaviour, and they do so only if the test rows resemble the rows the model will meet and were never used for any decision. A test set consulted ten times during development is a validation set with a misleading name.

How to apply

• Split once, early, before any exploration, and store the split (row identifiers or a seed with the exact splitting code) with the project.
• Use train_test_split(..., stratify=y) for classification so that rare classes appear in every part in the same proportion; the API documents the stratify parameter for this.
• Split by group (GroupKFold) when several rows come from one entity, such as one patient, one customer or one document, so that no entity appears on both sides.
• Split by time (TimeSeriesSplit or a fixed cut-off date) when the model will predict the future; a random split of time-ordered data trains on tomorrow and tests on yesterday.
• Evaluate on the test set once, at the end, and record the number together with the model version; if a further iteration is needed, treat the old test score as spent.

Pitfalls

Duplicates or near-duplicates across splits inflate scores. A tiny test set gives a point estimate with wide uncertainty; report an interval, not just the number. Cross-validation scores are model-selection scores and cannot replace the held-out test result.