Confidence-gated routing with a decision model: thresholds that scale with the stakes

Goal

Let an agent act automatically only when a decision is both clear and cheap to get wrong, and hand everything else to a confirmation step, a person or a stronger model.

Prerequisites

A decision expressed as a Choice or Score question. The documentation states that both carry a confidence between 0 and 1 derived from the shape of the returned probability distribution: all probability on one option gives 1.0, an even spread gives a low value. A Noul answer has no confidence; its own probability plays that role.

Steps

1. Enumerate the actions the answer can trigger and rank them by the cost of a wrong action (read-only display, reversible change, irreversible or external effect).
2. Set a global floor. The vendor's examples use 0.5 in one walkthrough and 0.6 in another; below the floor the agent never acts on the answer and instead asks, escalates or falls back.
3. Above the floor, give each action its own threshold: low for a read-only action, high for a destructive one. The confidence page shows the pattern in code: a balance lookup proceeds at any confidence above the floor, a transfer approval proceeds only above 0.9 and otherwise asks the user to confirm.
4. Keep the thresholds and the questions in one file so a reviewer can find them; the vendor's agent-skill page recommends exactly this for code written by coding agents.
5. Tune on labelled examples from your own traffic: for each threshold, count how many correct actions it blocks and how many wrong ones it lets through, and move it until the trade-off matches the stakes.
6. Pin the versioned model ID (jev-1.13.0 rather than jev-latest) once thresholds are tuned; the models page says aliases move with releases and that tuned thresholds should be re-checked on the new version before switching.
7. Record the answer, the probabilities and the confidence with every routed decision so that wrong actions can be traced to a threshold rather than guessed at.

Expected result

Automatic actions concentrate on clear cases; ambiguous inputs surface as confirmations or escalations rather than as silent wrong actions. Changing risk appetite means changing a constant, not a prompt.

Limits and test basis

Derived from the vendor's documentation; no threshold value here is a recommendation for any real system. The documentation itself says that correct thresholds depend on the domain and must be found on the caller's own data, and that confidence is a summary statistic, not a guarantee about the individual answer.

Check calibration before choosing thresholds

Before step 5, bucket the labelled answers by confidence (for example ten buckets of width 0.1) and compute the accuracy within each bucket. If accuracy rises with confidence and the top buckets reach the level the high-stakes action needs, thresholds on confidence are meaningful and can be placed where the buckets cross that level. If accuracy is flat across buckets, or the top bucket falls short of the required level, confidence is not carrying the information the gate needs on this data: keep the automatic path closed for that action and route it to a person or a stronger model regardless of the reported confidence. Repeat the check whenever the model version, the question wording or the input distribution changes.