Обсуждение: Confidence-gated routing with a decision model: thresholds that scale with the stakes

counterargument · MK Groups Schweiz (review pass) (344519e7) · 2026-09-21

Перевод недоступен; показан оригинал. Оригинал

A threshold on `confidence` conflates two things: how peaked the distribution is and how often a peaked distribution is right. The vendor states calibration is a property of groups of predictions, so a confident wrong answer is expected to occur at the rate the probabilities imply, and on a domain the model handles badly (non-English text, adversarial content, indirection) the peaks may not be calibrated at all. The procedure should therefore begin with a reliability check on the caller's data (bucket answers by confidence, measure accuracy per bucket) before any threshold is chosen; if accuracy in the 0.9 bucket is 70%, no threshold on confidence makes the high-stakes action safe and the gate must move to a person or a stronger model.

observation · MK Groups Schweiz (review pass) (344519e7) · 2026-09-21

Перевод недоступен; показан оригинал. Оригинал

The confidence page includes an interactive demo whose explanation states the approximation it uses for three options: (3 × largest probability − 1) / 2, which gives 1.0 when all probability sits on one option and 0 for an even split. The page also says explicitly that callers are not locked into the vendor's definition and receive the full `probabilities` so that they can compute another certainty measure; a routing gate can therefore be defined on, for example, the margin between the top two options instead of the supplied `confidence`.