Invalidating test evidence when a later edit changes what was checked

Goal

Keep a coding agent from carrying a passing result forward after edits have changed the code, configuration, fixtures, or dependencies that result actually covered.

Prerequisites

Use a compact validation record containing the checked revision or file fingerprints, command, environment identity, result, and relevant generated artifacts. The record may be temporary; do not preserve secrets or full test data.

Steps

1. Before a check, identify its inputs at the level needed for the decision. Include configuration and generated files when they affect execution; recording only the edited source filename is insufficient for a reproducible claim.

2. Attach the completed result to those inputs. Distinguish a check that was started from one that finished, and retain the actual result rather than the expected success state.

3. After each later edit, determine which evidence depends on the changed inputs. Mark affected results stale immediately, including results from another agent that ran against an earlier shared checkout.

4. Rerun the smallest set of checks that resolves the resulting uncertainty, following project requirements. A documentation-only change may leave runtime evidence applicable, but record the reasoning when the boundary is unclear.

5. Before reporting completion, compare the submitted artifact with the validated artifact. Exercise the bookkeeping with a deliberate source edit after a passing check and verify that the final report refuses to treat the old result as current.

Expected result

Every claimed passing check identifies the artifact it covered. Reviewers can distinguish current validation, still-relevant earlier evidence, and stale results that require another run.

Limits and test basis

This is a proposed evidence discipline with no measured effectiveness claim. Dependency mapping may be incomplete, so uncertainty can justify broader checks. A current passing result still proves only the behaviors exercised by that check.