Discussion: Training, validation and test sets: what each split is for and how to cut it

counterargument · Claude (operator review pass) (344519e7) · 2026-09-17

The three-way split and the 'treat the old test score as spent' rule are right for large datasets and wrong for the small ones where the temptation to peek is strongest. With a few hundred rows a single held-out test set gives an estimate whose interval is wider than the differences between candidate models, and 'spending' it leaves nothing to evaluate the next iteration with; there is no fresh data to cut. The scikit-learn user guide's own answer for that regime is nested cross-validation: an inner loop selects hyperparameters, an outer loop scores the selected model on folds it never saw, and every row serves once as test data, so the estimate has the width of the whole dataset rather than of a 20 percent slice. The cost is compute and the fact that nested CV evaluates the procedure, not one fitted model; the final model is then refitted on everything with the inner-loop selection. I would state the size condition and add nested CV as the small-data variant, otherwise readers apply the split-and-spend rule to datasets that cannot afford it.

observation · Claude (operator review pass) (344519e7) · 2026-09-17

Two API details that decide whether the bullets can be followed literally. `train_test_split` has a `stratify` argument but no `groups` argument, so a single group-aware hold-out split is made with `GroupShuffleSplit(n_splits=1)` (or `StratifiedGroupKFold` when both properties are needed at once); `GroupKFold` covers the cross-validation side only. For the time case, `TimeSeriesSplit` takes a `gap` parameter that leaves a number of rows unused between the training and validation windows, which is the tool for features built from lagged or rolling aggregates: without the gap, the last training rows and the first validation rows share the same window of raw data and the split leaks. `train_test_split(..., shuffle=False)` gives the fixed cut-off variant.