Auditing the candidate generator before judging Jev selection quality

Goal

Measure whether the correct option reached a decision request before attributing a wrong selection to the model or trying to repair it with stronger wording.

Prerequisites

Prepare labelled examples with known acceptable outcomes, the candidate-generation version, and the exact candidate set offered for each case. Define how equivalent answers and genuinely missing answers are represented.

Steps

1. For every example, determine whether at least one acceptable candidate was present. Keep this availability label independent of the model response so a confident wrong selection cannot conceal an upstream omission.

2. Separate cases into absent-candidate failures, selection failures among available candidates, and downstream mapping failures. Inspect these groups with different owners and corrective actions.

3. Vary the shortlist construction using the same evaluation material. For repository navigation, include alternate symbol names and moved files; for document extraction, include relevant spans missed by the original pattern.

4. Keep an explicit unresolved route when the candidate list may be incomplete. Reopening retrieval is a separate bounded action, not an instruction to repeatedly choose from the same inadequate list.

5. Report both end-to-end success and performance conditional on an acceptable candidate being present. Review examples excluded from the latter figure so that the headline does not hide the retrieval problem.

Expected result

The evaluation identifies where effort belongs: inventory coverage, retrieval, selection, or action mapping. A model comparison then states the candidate conditions under which its results were obtained.

Limits and test basis

This is a proposed evaluation method, not a reported benchmark. TypeSafe documents decisions over supplied options and code-owned workflow. Better conditional selection quality does not compensate for missing options in the real workload. The underlying interface or pattern is described in How to build with TypeSafe; the workflow above is a proposed adaptation.