Data leakage in machine learning: how information from the future or the test set gets into a model

What it is

The scikit-learn pitfalls page defines data leakage as using information when building the model that would not be available at prediction time, and states that the result is an overly optimistic performance estimate followed by poorer performance on genuinely new data. Its general rule: test data should never be used to make choices about the model, and fit is never called on the test data, including the fit of preprocessing steps such as scalers, imputers and encoders. The page recommends a Pipeline so that every step is fitted on the training fold only, also inside cross-validation.

Why it matters

Leakage does not raise an error. The validation score looks excellent, the model is shipped, and the gap appears only when real predictions are compared with real outcomes weeks later. By then the offline number has been quoted in decisions.

How to apply

• Put every data-dependent transformation (scaling, imputation, encoding, feature selection, resampling) inside the pipeline that is cross-validated, never before the split.
• Audit each feature for target leakage: a column that is filled in after the outcome is known (a "refund issued" flag when predicting churn, a diagnosis code when predicting admission) predicts the target perfectly and is useless at prediction time.
• Check timestamps: every feature value must be computable from data that existed at the moment the prediction would have been made. Aggregates such as "total purchases" must be cut off at that moment.
• Keep rows of one entity on one side of the split with group-aware splitters such as GroupKFold, which the cross-validation guide describes for exactly this case.
• Be suspicious of scores far above the baseline or above what domain experts consider possible; leakage is the first hypothesis to test.

Pitfalls

Deduplication after the split leaves near-duplicates on both sides. Feature selection on the whole dataset before cross-validation leaks the target through the selected columns. Target encoding fitted on the same rows it encodes leaks the label into the feature.