Feature scaling and categorical encoding: what to transform, and fit it on training data only

What it is

The scikit-learn preprocessing guide states that standardisation (subtracting the mean, dividing by the standard deviation, StandardScaler) is a common requirement for many estimators, which may behave badly when features are not roughly centred with unit variance; MinMaxScaler maps to a fixed range and RobustScaler uses more robust estimates of centre and range (by default the median and the interquartile range) for data with many outliers. For categories, OneHotEncoder turns a column with n values into n binary columns, OrdinalEncoder assigns integers (meaningful only when the order is real), and TargetEncoder uses the target mean conditioned on the categorical value, which the guide describes as useful for high-cardinality features where one-hot columns would inflate the feature space. The pitfalls page adds the rule that governs all of them: never call fit on test data; the transformer learns its statistics from the training rows and is then applied to everything else.

Why it matters

Models that use distances (k-nearest neighbours, SVMs, k-means), coefficient penalties (ridge, lasso, logistic regression) or gradient descent (neural networks) treat a feature in metres and one in millimetres as different in importance by a factor of a thousand. Encoding decides whether a model can use a category at all and whether a rare category breaks inference.

How to apply

• Tree-based models are invariant to monotone scaling; scale for everything else, and always when a penalty is applied.
• One-hot for low cardinality and linear models; ordinal only for genuinely ordered levels; target encoding for many levels, using fit_transform, which the guide states relies on an internal cross-fitting scheme to keep the target from leaking into the training representation.
• Set handle_unknown on encoders so that a category first seen in production yields a defined output rather than an exception.
• Wrap scaler, encoder and model in one Pipeline (with ColumnTransformer for mixed columns) so that fit and transform cannot be applied to the wrong rows.
• Persist the fitted pipeline as one artefact; a model served with a different scaler than it was trained with is a silent bug.

Pitfalls

Scaling before the split leaks test statistics into training. Ordinal codes on nominal categories invent an order the model will exploit. Log-transforming a feature with zeros or negatives fails; use log1p or a shift, and document it.