Building an evaluation harness for agent tasks

Goal

Decide whether a change to a prompt, tool set or model makes an agent better or worse on the tasks it is actually used for, with a number that can be compared across runs rather than an impression from a few transcripts.

Prerequisites

A task list with a checkable end state per task (a database row, a file, a returned value); an environment that can be reset between runs (container, fixture database, recorded HTTP responses); a budget for repeated runs. τ-bench is a documented example of the shape: the agent works with domain tools and policy rules against a simulated user, and the paper evaluates by comparing the database state at the end of the conversation with an annotated goal state. Frameworks such as Inspect (UK AI Security Institute) describe an evaluation as composable building blocks (datasets, agents, tools and scorers) and provide sandboxing for untrusted model code and tool approval.

Steps

1. Collect tasks from real usage: failed sessions, support tickets, edge cases, not only easy successes. Write each as an input plus an oracle (expected end state or a deterministic check), never as an expected transcript.
2. Freeze the environment: pin tool versions, seed data and recorded external responses, so that a failing run can be re-run with identical inputs.
3. Grade the end state in code. Use a model-graded rubric only where no deterministic check exists, and calibrate it against a sample of human labels.
4. Run every task k times. The τ-bench paper proposes pass^k, the probability that all k trials succeed, next to the usual pass@k; report both, since the gap between them is the agent's inconsistency.
5. Record per-run cost, steps and tokens with the score, so that an improvement that doubles cost is visible.
6. Keep a held-out set that is never used while tuning prompts; report tuning and held-out results separately.
7. Store the configuration (model, prompt version, tool set, seed) with each result and run the harness on every prompt or model change.

Expected result

A score per task and per run with a confidence band, a list of tasks that flip between pass and fail across trials, and a cost per completed task, all reproducible from the stored configuration.

Limits and test basis

The cited paper reports that state-of-the-art function-calling agents at the time succeeded on under 50% of its tasks and were inconsistent (pass^8 below 25% in its retail domain); no figures for other agents or tasks are claimed here. Model-graded scoring inherits the grader's biases, small task sets give wide confidence bands, and the harness measures only what its tasks cover.