A first game day: one chaos experiment with a hypothesis, a blast radius and an abort rule

Goal

Learn, on a chosen day with everyone watching, what happens when one component fails, instead of learning it at night; and rehearse the human response (alerts, runbooks, roles) at the same time.

Prerequisites

Monitoring that shows the steady-state signal in near real time; a tested undo for the fault; a change-calendar entry and, if production is involved, a status-page note; a facilitator who is not one of the responders.

Steps

1. Choose one fault that is plausible and reversible: stop one replica, block one dependency's port, fill a queue, expire a cache, revoke one credential in staging. Not "take down the database".
2. Define the steady state as, in the words of the Principles of Chaos Engineering, some measurable output of the system that indicates normal behaviour (request success rate, completed checkouts per minute), and write the hypothesis: "this signal stays in its normal band while replica 2 is stopped".
3. Write the expected system behaviour and the expected human behaviour separately: which alert fires and within how long, which runbook is used, who is paged.
4. Set the blast radius (one replica, one region, a fraction of traffic) and the abort condition (the signal leaves its band for more than a set number of minutes, or any customer-visible error) with the undo command ready in a terminal.
5. Announce the start, inject the fault, start a timer. The facilitator records timestamps: fault injected, first alert, first human action, mitigation, recovery. Responders act as on a real page; the facilitator answers no question the monitoring could answer.
6. Undo the fault at the planned end or on abort; verify the steady state; confirm no lingering effects (queues drained, connections reset, alerts cleared).
7. Debrief within the hour: was the hypothesis disproved? Which alert did not fire, which runbook step was wrong, which dashboard was missing? Write action items with owners as for a postmortem.
8. If a real injection is not yet acceptable, run the same scenario as a tabletop first. The SRE book describes "Wheel of Misfortune": a game master presents a scenario and the on-call pair narrates what they would do, with the group learning from each round.

Expected result

A written record of one fault's actual effect, a short list of gaps found, and a team that has practised declaring and handling an incident without a real outage.

Limits and test basis

A staging environment proves less than production, and the Principles argue for production because behaviour differs by environment and traffic; a first run in staging is still the safer start. The protocol is a synthesis of the cited sources; no finding from running it is claimed.