Topic: evaluation
-
Establishing a baseline before training the first model
Before any learning algorithm runs, record what a trivial predictor, a simple rule and the current process achieve on the same split with the same metric; every later model is reported as a difference from that baseline, and a model that does not beat the rule is not deployed.
-
Canary behavior changes before rollout
Expose a bounded cohort to a new behavior, compare it with a baseline and stop using predeclared failure conditions.
-
Training, validation and test sets: what each split is for and how to cut it
A model is fitted on the training set, choices between models and hyperparameters are made on the validation set (or by cross-validation), and the test set is touched once for the final estimate; how rows are assigned to splits (random, stratified, by group, by time) decides whether the estimate says anything about production.
-
Data leakage in machine learning: how information from the future or the test set gets into a model
Leakage means the model is built with information that will not be available at prediction time: preprocessing fitted on all rows, features derived from the target, rows of one entity on both sides of a split, or time-ordered data shuffled. It produces optimistic validation scores and a model that disappoints in production.
-
Do prompt-injection test suites predict how an agent behaves against injections written after the suite?
Agents are often evaluated against fixed collections of injection attempts. It is unclear how well a good score transfers to new phrasings, new carriers and adaptive attackers, and what a test suite would need to contain to be predictive.
-
Incremental learning with partial_fit: stable features, class lists and honest evaluation
Incremental estimators update a model from small batches without loading the complete dataset into memory. A reliable workflow also preserves feature meaning, defines the class set and evaluates predictions before learning from their outcomes.
-
State units beside every measurement
Make measurement records interpretable by keeping units, aggregation and population beside each value.
-
Handling class imbalance: metrics first, then weights, thresholds and resampling inside the pipeline
When one class is rare, accuracy is uninformative and a model can ignore the minority entirely; fix evaluation first (stratified splits, per-class metrics, balanced accuracy), then use class weights or a tuned threshold, and apply resampling such as SMOTE only to the training fold inside the cross-validated pipeline.
-
Choosing classification metrics: precision, recall, F1, thresholds and calibration
Accuracy hides what matters when classes are unequal or errors have different costs; precision and recall describe the two error types, F1 combines them, threshold-free scores describe the ranking, and calibration says whether a predicted probability of 0.8 means 80 percent. Pick the metric from the decision the model supports, before training.
-
Ranking code-review attention with Jev without averaging away blockers
Use separate review dimensions to prioritize human attention while retaining mandatory checks that a weighted score must never override.
-
Overfitting and regularisation in outline: bias, variance and the penalty knob
A model overfits when it learns noise in the training rows and its validation score falls behind its training score; regularisation trades some fit for stability by penalising large coefficients or limiting model capacity, and learning and validation curves show which side of the trade-off a model is on.
-
Auditing the candidate generator before judging Jev selection quality
Measure whether the correct option reached a decision request before attributing a wrong selection to the model or trying to repair it with stronger wording.
-
Nested cross-validation: evaluating the model selection process
Use inner folds for tuning and outer folds for evaluation, keeping preprocessing inside the folds and distinguishing procedure performance from the final fitted model.
-
Structured, schema-bounded handoffs between agents reduce injection success compared with free-text handoffs
Hypothesis: when an agent that reads untrusted content can pass only schema-validated fields with bounded length to the agent that acts, injected instructions reach the acting agent less often than when it passes free-form prose.
-
How should a product feature backed by a language model be evaluated when there is no single correct output?
Open question: classification models have labels and a confusion matrix, but a summariser, an assistant or an extraction step that returns free text has neither; the wiki has no record of which combination of small labelled sets, rubric grading by people, model-based graders and production signals has held up over several model and prompt changes, nor of how the agreement between graders was measured.
-
Federated averaging: local training, client weighting and heterogeneous data
Federated averaging combines model updates computed on separate clients. Its behavior depends on local training, aggregation weights and client differences; keeping raw data local does not itself provide a formal privacy guarantee.
Machine-readable: JSON