Diffing the OpenAPI document in CI catches breaking changes that code review misses

Hypothesis

Breaking changes to an HTTP API are often invisible in the code diff a reviewer reads: a renamed struct field, a validator made stricter, a serialiser that stops emitting a property. They are visible in the generated API description. The OpenAPI specification (cited) describes the document as letting humans and computers understand a service's capabilities without access to source code; a tool such as oasdiff (cited as a command-line tool to compare and detect breaking changes in OpenAPI specs) classifies the differences between two documents as breaking or not. Google's AIP-180 on backwards compatibility (cited) lists which changes count as breaking, such as removing or renaming fields and changing types. The hypothesis is that gating merges on such a diff catches a class of regressions that review of the implementation alone does not, with a tolerable false-positive rate once volatile parts of the document are excluded.

Prediction

Repositories that add the gate will record, per quarter, fewer breaking changes discovered after release (client bug reports, rollbacks, emergency compatibility patches) than in the quarters before, and the gate's log will contain blocked breaking changes that the corresponding pull request did not mention. The number of intended breaking changes will not fall; they will instead be labelled and routed through the versioning process.

Proposed test

1. Choose services that generate their OpenAPI document from code, so the document reflects the implementation, and that have at least two quarters of release history.
2. Reconstruct from git history the document at each release, run the diff retrospectively, and match changes classified as breaking against known post-release incidents.
3. Turn the gate on; for each blocked change record whether the author intended it, whether review had noticed it, and how it was resolved.
4. Compare post-release breaking-change incidents per release before and after; report false positives (flagged but harmless) as a separate count.

Status

No result is claimed. The gate cannot see behavioural changes the document does not express (meaning of a value, ordering, timing), and its classification rules may differ from what a particular client base tolerates.