Calibrating a scanner result with a vulnerable fixture and a safe twin

Goal

Determine whether a security scanner distinguishes the behavior it claims to detect. This original method uses controlled fixtures to interpret a finding, not to certify the scanner or rank products.

Prerequisites

Use a disconnected or otherwise access-restricted lab owned by the operator. Prepare a deliberately flawed toy case and a corrected twin with the same surrounding structure. Keep real credentials, personal data, and production targets out.

Steps

1. State the exact condition the scanner is supposed to report and the evidence that would establish it. Avoid an oracle that simply repeats the scanner’s rule identifier or severity label.

2. Run the selected rule against the flawed fixture and preserve minimal diagnostic metadata. If no result appears, first verify that the fixture was included in the scan and that the rule was enabled.

3. Run the corrected twin with equivalent configuration. A finding on both cases requires investigation into rule precision, fixture differences, or a remaining flaw; it is not automatically a false positive.

4. Change one relevant property at a time, such as the data-flow boundary the rule models. Keep unrelated build, dependency, and scan configuration stable across the comparison.

5. Record the actual outputs and the independent oracle judgment separately. When adding a suppression, attach the bounded reason and a condition that should trigger re-evaluation.

Expected result

A usable assessment distinguishes detection failure, false alarm, and unresolved evidence. It also states which fixture and scanner configuration the judgment covers.

Limits and test basis

Passing these controls does not estimate real-world recall or precision. This article reports no scanner execution, measurements, or observed findings; it proposes a reproducible local comparison. This is an original proposed method; no execution or empirical result is claimed.