Selecting a tool or skill with a decision model: Choice to rank, Noul to abstain

What it is

An agent with many tools or skills has to decide, each turn, whether any of them applies and which one. TypeSafe's skill-suggestion cookbook does this over the 182 skills of Nous Research's Hermes catalogue in two requests: the first ranks every skill with one Choice (whose 255-option limit bounds the catalogue size per question) and asks with a Noul whether the turn needs a skill at all; the second reads the top three candidates with their full descriptions and can reject all of them. Only a winner's name goes into the agent's system prompt.

Why it matters

The two question types answer different questions. The documentation's failure-mode page states it directly: a Choice is relative and settles which option, while each Noul is absolute and can be low for all options. A Choice alone always names a winner, even when nothing fits; a set of Nouls alone can say that several fit without ordering them. Tool selection needs both: an ordering and a licence to pick nothing.

How to apply

• Give the catalogue as state with short, distinct descriptions per candidate and ask one Choice over the candidate keys; keep an explicit "none" option even when a separate Noul asks whether a tool is needed, so the two signals can be compared.
• Ask the abstain question as a Noul with the yes reading as the natural one ("this turn needs a tool") and choose its threshold separately from the Choice's confidence; thresholds do not transfer between types.
• For the shortlist step, send only the top candidates with their full documentation; the first step's descriptions must be short because all of them share one state budget.
• Keep the final decision in code: rank by Choice probability, require the Noul above its threshold, and fall back to no tool when either fails.
• Record which candidates were shortlisted and why, so that a missing tool in the catalogue shows up as a pattern of "none" answers.

Pitfalls

Descriptions that overlap produce flat distributions and low confidence that looks like ambiguity in the turn but is ambiguity in the catalogue; fix the descriptions first. Candidate names are not sent to the model as keys, so the meaning must be in the description. A catalogue above 255 entries needs a hierarchy or a first-stage filter, which the vendor's hierarchical-classification cookbook addresses with beam search over Choice probabilities.