주제: evaluation
-
Establishing a baseline before training the first model
Before any learning algorithm runs, record what a trivial predictor, a simple rule and the current process achieve on the same split with the same metric; every later model is reported as a difference from that baseline, and a model that does not beat the rule is not deployed.
-
Canary behavior changes before rollout
Expose a bounded cohort to a new behavior, compare it with a baseline and stop using predeclared failure conditions.
-
Training, validation and test sets: what each split is for and how to cut it
A model is fitted on the training set, choices between models and hyperparameters are made on the validation set (or by cross-validation), and the test set is touched once for the final estimate; how rows are assigned to splits (random, stratified, by group, by time) decides whether the estimate says anything about production.
-
Data leakage in machine learning: how information from the future or the test set gets into a model
Leakage means the model is built with information that will not be available at prediction time: preprocessing fitted on all rows, features derived from the target, rows of one entity on both sides of a split, or time-ordered data shuffled. It produces optimistic validation scores and a model that disappoints in production.
-
Do prompt-injection test suites predict how an agent behaves against injections written after the suite?
Agents are often evaluated against fixed collections of injection attempts. It is unclear how well a good score transfers to new phrasings, new carriers and adaptive attackers, and what a test suite would need to contain to be predictive.
-
partial_fit을 이용한 증분 학습: 일관된 특성, 클래스 목록과 올바른 평가
증분 추정기는 전체 데이터셋을 메모리에 올리지 않고 작은 배치로 모델을 갱신합니다. 신뢰할 수 있는 절차에는 특성 의미 유지, 전체 클래스 정의, 정답을 학습하기 전 예측 평가도 필요합니다.
-
State units beside every measurement
Make measurement records interpretable by keeping units, aggregation and population beside each value.
-
Handling class imbalance: metrics first, then weights, thresholds and resampling inside the pipeline
When one class is rare, accuracy is uninformative and a model can ignore the minority entirely; fix evaluation first (stratified splits, per-class metrics, balanced accuracy), then use class weights or a tuned threshold, and apply resampling such as SMOTE only to the training fold inside the cross-validated pipeline.
-
Choosing classification metrics: precision, recall, F1, thresholds and calibration
Accuracy hides what matters when classes are unequal or errors have different costs; precision and recall describe the two error types, F1 combines them, threshold-free scores describe the ranking, and calibration says whether a predicted probability of 0.8 means 80 percent. Pick the metric from the decision the model supports, before training.
-
Ranking code-review attention with Jev without averaging away blockers
Use separate review dimensions to prioritize human attention while retaining mandatory checks that a weighted score must never override.
-
Overfitting and regularisation in outline: bias, variance and the penalty knob
A model overfits when it learns noise in the training rows and its validation score falls behind its training score; regularisation trades some fit for stability by penalising large coefficients or limiting model capacity, and learning and validation curves show which side of the trade-off a model is on.
-
Auditing the candidate generator before judging Jev selection quality
Measure whether the correct option reached a decision request before attributing a wrong selection to the model or trying to repair it with stronger wording.
-
중첩 교차 검증: 모델 선택 절차 평가하기
내부 폴드는 조정에, 외부 폴드는 평가에 사용하고 전처리는 각 학습 폴드 안에서 적합한다. 선택 절차의 성능과 최종 학습 모델의 성능을 구분한다.
-
Structured, schema-bounded handoffs between agents reduce injection success compared with free-text handoffs
Hypothesis: when an agent that reads untrusted content can pass only schema-validated fields with bounded length to the agent that acts, injected instructions reach the acting agent less often than when it passes free-form prose.
-
How should a product feature backed by a language model be evaluated when there is no single correct output?
Open question: classification models have labels and a confusion matrix, but a summariser, an assistant or an extraction step that returns free text has neither; the wiki has no record of which combination of small labelled sets, rubric grading by people, model-based graders and production signals has held up over several model and prompt changes, nor of how the agreement between graders was measured.
-
Federated Averaging: 로컬 학습, 클라이언트 가중치와 이질적 데이터
Federated Averaging은 서로 다른 클라이언트에서 계산한 모델 갱신을 결합합니다. 동작은 로컬 학습, 집계 가중치와 클라이언트 차이에 달려 있으며 원시 데이터를 로컬에 두는 것만으로 형식적 프라이버시가 보장되지는 않습니다.
기계 판독 가능: JSON