テーマ： machine-learning
-
Establishing a baseline before training the first model
Before any learning algorithm runs, record what a trivial predictor, a simple rule and the current process achieve on the same split with the same metric; every later model is reported as a difference from that baseline, and a model that does not beat the rule is not deployed.
-
Feature scaling and categorical encoding: what to transform, and fit it on training data only
Distance- and gradient-based models need numeric features on comparable scales (StandardScaler, MinMaxScaler, RobustScaler); categorical columns become numbers by one-hot, ordinal or target encoding depending on cardinality and model type. Every transformer is fitted on the training split and applied unchanged to validation, test and production rows.
-
Reproducibility of a machine-learning experiment: seeds, environment, data and the limits of determinism
Rerunning an experiment and getting the same number requires fixed random states passed explicitly, pinned library versions, an identified dataset and split, and awareness that GPU kernels and library releases can still change results; the protocol makes runs repeatable where possible and documents where they are not.
-
Training, validation and test sets: what each split is for and how to cut it
A model is fitted on the training set, choices between models and hyperparameters are made on the validation set (or by cross-validation), and the test set is touched once for the final estimate; how rows are assigned to splits (random, stratified, by group, by time) decides whether the estimate says anything about production.
-
Data leakage in machine learning: how information from the future or the test set gets into a model
Leakage means the model is built with information that will not be available at prediction time: preprocessing fitted on all rows, features derived from the target, rows of one entity on both sides of a split, or time-ordered data shuffled. It produces optimistic validation scores and a model that disappoints in production.
-
Handling class imbalance: metrics first, then weights, thresholds and resampling inside the pipeline
When one class is rare, accuracy is uninformative and a model can ignore the minority entirely; fix evaluation first (stratified splits, per-class metrics, balanced accuracy), then use class weights or a tuned threshold, and apply resampling such as SMOTE only to the training fold inside the cross-validated pipeline.
-
Choosing classification metrics: precision, recall, F1, thresholds and calibration
Accuracy hides what matters when classes are unequal or errors have different costs; precision and recall describe the two error types, F1 combines them, threshold-free scores describe the ranking, and calibration says whether a predicted probability of 0.8 means 80 percent. Pick the metric from the decision the model supports, before training.
-
Embeddings as a data type: fixed-length vectors, a distance function and what a column of them needs
An embedding is a fixed-length float vector produced by a specific model, meaningful only under the distance that model was trained for and only next to vectors from the same model version; storing it requires the dimension, the model identifier and the distance to be recorded, and querying it means nearest-neighbour search, exact or approximate.
-
Monitoring a deployed model for drift: inputs, outputs and delayed labels
A model's offline score stops being true the moment the input distribution, the label distribution or the relationship between them changes; monitor feature and prediction distributions against a training reference, log served features to detect training-serving skew, and join delayed labels back to compute the real metric with a lag.
-
Overfitting and regularisation in outline: bias, variance and the penalty knob
A model overfits when it learns noise in the training rows and its validation score falls behind its training score; regularisation trades some fit for stability by penalising large coefficients or limiting model capacity, and learning and validation curves show which side of the trade-off a model is on.
-
Versioning a trained model: the artefact together with the code, data, parameters and environment that produced it
A model file alone cannot be reproduced, audited or safely replaced; version it as a record that links the serialised artefact to the code commit, the dataset version, the hyperparameters, the metrics on the fixed test split and the exact dependency versions, and promote versions with aliases rather than by overwriting a file.
-
How should a product feature backed by a language model be evaluated when there is no single correct output?
Open question: classification models have labels and a confusion matrix, but a summariser, an assistant or an extraction step that returns free text has neither; the wiki has no record of which combination of small labelled sets, rubric grading by people, model-based graders and production signals has held up over several model and prompt changes, nor of how the agreement between graders was measured.
機械可読： JSON