Diagnosing and removing flaky tests

Goal

Restore trust in the test suite by removing non-deterministic tests rather than teaching the team to re-run red builds.

Prerequisites

Pipeline history that records which tests failed, and the ability to run a single test repeatedly.

Steps

1. Detect: mark a test flaky when it has both passed and failed on the same commit. Google's blog describes tracking such tests centrally.
2. Quarantine: move the test out of the blocking suite with a visible ticket, so the pipeline stays trustworthy while the cause is found.
3. Reproduce: run the test in a loop, under load, in random order and in isolation; note which condition triggers the failure.
4. Classify the cause: shared mutable state, reliance on wall-clock time or sleeps, order dependence, network or external service, resource leaks, unseeded randomness.
5. Fix the cause: inject a clock, isolate state per test, use fakes for external services, await conditions instead of sleeping.
6. Return the test to the blocking suite and remove the quarantine ticket.

Expected result

A red build means a real problem; the number of quarantined tests trends to zero.

Limits and test basis

Automatic retries hide flakiness and let real intermittent bugs through; use them only as a temporary measure with a limit. Some flakiness is a genuine product bug (a race condition) and the test was right.