Sujet : evaluation
-
Établir une référence avant d’entraîner le premier modèle
Avant d’exécuter un algorithme d’apprentissage, consigner les résultats d’un prédicteur trivial, d’une règle simple et du processus actuel sur la même partition, avec la même métrique. Chaque modèle ultérieur est présenté par son écart à cette référence, et un modèle qui ne fait pas mieux que la règle n’est pas déployé.
-
Canary behavior changes before rollout
Expose a bounded cohort to a new behavior, compare it with a baseline and stop using predeclared failure conditions.
-
Training, validation and test sets: what each split is for and how to cut it
A model is fitted on the training set, choices between models and hyperparameters are made on the validation set (or by cross-validation), and the test set is touched once for the final estimate; how rows are assigned to splits (random, stratified, by group, by time) decides whether the estimate says anything about production.
-
Data leakage in machine learning: how information from the future or the test set gets into a model
Leakage means the model is built with information that will not be available at prediction time: preprocessing fitted on all rows, features derived from the target, rows of one entity on both sides of a split, or time-ordered data shuffled. It produces optimistic validation scores and a model that disappoints in production.
-
State units beside every measurement
Make measurement records interpretable by keeping units, aggregation and population beside each value.
-
Handling class imbalance: metrics first, then weights, thresholds and resampling inside the pipeline
When one class is rare, accuracy is uninformative and a model can ignore the minority entirely; fix evaluation first (stratified splits, per-class metrics, balanced accuracy), then use class weights or a tuned threshold, and apply resampling such as SMOTE only to the training fold inside the cross-validated pipeline.
-
Choosing classification metrics: precision, recall, F1, thresholds and calibration
Accuracy hides what matters when classes are unequal or errors have different costs; precision and recall describe the two error types, F1 combines them, threshold-free scores describe the ranking, and calibration says whether a predicted probability of 0.8 means 80 percent. Pick the metric from the decision the model supports, before training.
-
Overfitting and regularisation in outline: bias, variance and the penalty knob
A model overfits when it learns noise in the training rows and its validation score falls behind its training score; regularisation trades some fit for stability by penalising large coefficients or limiting model capacity, and learning and validation curves show which side of the trade-off a model is on.
-
How should a product feature backed by a language model be evaluated when there is no single correct output?
Open question: classification models have labels and a confusion matrix, but a summariser, an assistant or an extraction step that returns free text has neither; the wiki has no record of which combination of small labelled sets, rubric grading by people, model-based graders and production signals has held up over several model and prompt changes, nor of how the agreement between graders was measured.
Lisible par machine : JSON