Keeping an agent-generated test expectation independent of the implementation

Goal

Check whether a proposed test verifies the requirement or simply repeats the same mistaken assumption used to write the implementation.

Prerequisites

Have an explicit behavior requirement, the candidate implementation, and a proposed test. Identify who can resolve ambiguous expected outcomes before treating a generated fixture as authoritative.

Steps

1. Derive the expected result from the requirement before examining the implementation’s output. Record the relevant rule in plain language so the expected value has an origin independent of the code under test.

2. For computed expectations, use a simpler independent method or a hand-checkable example. Avoid copying the same helper, branch structure, or parsing rule into both production code and the test oracle.

3. Include a case that distinguishes the intended rule from a plausible wrong interpretation. For a boundary rule, choose values on the disputed boundary rather than only ordinary examples on which both interpretations agree.

4. Run the test against a deliberately incorrect local variant when appropriate and authorized. If the assertion still passes, inspect whether the fixture reaches the changed behavior and whether the assertion observes the relevant output.

5. Review any fixture updates after the implementation changes. A changed output is not automatically a new expected result; require a requirement change or evidence that the earlier expectation was wrong.

Expected result

The test carries an explainable expectation and a case capable of rejecting at least the identified alternative behavior. Reviewers can assess its value without trusting the agent’s implementation as its own specification.

Limits and test basis

This is a proposed test-review method with no test run claimed. Independent oracles can still share incorrect requirements. Deliberately altered variants belong in isolated test work and must not be left in the submitted implementation.