What share of accessibility defects found in manual audits or by users had passed the automated checks in CI, and which kinds escaped?

Open question

Automated accessibility checkers are cheap to run: a rule engine inspects the rendered DOM on every build and fails the pipeline on missing alternative text, low contrast, form controls without labels or invalid ARIA attributes. The W3C WAI guidance on selecting evaluation tools is explicit that tools save time but cannot do everything, because some accessibility checks cannot be automated and require manual intervention. The temptation is nevertheless to treat a green automated run as "accessibility done", because a manual audit costs money and user reports arrive through support channels rather than the build.

What the wiki lacks is a record of the gap on real sites. When a manual audit (an expert review with a screen reader and keyboard, or a session with disabled users) was carried out on a site whose builds already passed an automated checker, what share of the defects found had been present in the checked pages and passed anyway? Which defect types dominate that share: focus order and focus loss, keyboard traps in custom widgets, meaningless link text that is technically labelled, wrong reading order, status messages that are never announced, alternative text that is present but wrong, motion that cannot be paused? And the reverse: what share of the automated findings turned out to be irrelevant to any user of the site, so that the team learned to ignore the tool? Does the gap shrink when the automated run includes scripted keyboard interaction, or when the component library was itself audited, so that the remaining defects live in page composition rather than in components?

The answer decides where a small team's limited accessibility effort should go: more rules in CI, a periodic manual pass, or a design-system audit.

What a useful answer contains

The site type, the framework and component library, and the checker with its rule set and version. The number of pages covered by the automated run and by the manual audit. Per defect found manually or reported by a user: the success criterion concerned, whether the page had passed the automated run, and whether a rule for it exists in the tool but did not fire. Per automated finding over the same period: whether it was fixed, suppressed or judged a false positive, with the reason. The audit method (expert review, assistive-technology testing, user session), since each finds different things. Whether the team changed anything as a result: added rules, added a manual checklist to the review, or replaced components. Single-site reports are welcome if the rule set and audit method are stated; comparisons across sites with the same tool are more useful.