How should the reliability of an acting agent be measured when a run can succeed at its task and still cause an unwanted side effect?

Open question

The cited τ-bench paper scores an agent by comparing the database state at the end of a conversation with an annotated goal state and proposes pass^k for reliability over repeated trials. Both measure whether the goal was reached. An agent that acts on real systems can reach the goal and still do something it should not have: touch a file outside the task, run a command with a wider effect than needed, send a message that was not asked for, retry a non-idempotent call, or consume a cost budget several times over. A task-success metric counts such a run as a pass; an incident count in production counts only the cases someone noticed. What is missing is a documented practice for measuring the second dimension:

• What is counted as an unwanted side effect, and by whom: a fixed list of forbidden effects per task, a diff of the environment against an expected diff, an approval gate's denial log, or a human review of the action log?
• How the two dimensions are reported together: as a pair (task success, clean-run rate), as a single score with a penalty, or as pass^k over runs that are both successful and clean?
• How stable the side-effect rate is across prompt versions, model versions and repeated runs, and whether it moves in the same direction as task success or independently.
• What share of side effects is caught by dry-run and approval mechanisms before they happen, and what share only by inspection afterwards.
• Whether environments with a full pre/post state diff (containers, snapshots) have produced counts that differ from self-reported action logs.

What a useful answer contains

The agent's task type and tool set; the definition of an unwanted side effect used, with examples of what was and was not counted; the observation method (environment diff, log review, gate denials); the number of runs; task success and side-effect counts side by side, per version if several were compared; and what the team changed as a result. Proposals without data should be labelled as proposals, and single-team reports should say so.