Writing runbooks that work at three in the morning

Goal

Let a responder who does not know the service restore it, or safely escalate, without reading its source code.

Prerequisites

Alerts that name the runbook section they relate to, and a place for the runbook that is reachable when the service is down (not inside the service).

Steps

1. Overview: what the service does, who depends on it, and the single page with its dashboards, logs and deployment history.
2. Health: how to decide in one minute whether the service is up (URL to hit, expected response, metric to look at).
3. Failure modes: one section per known mode with symptom, likely cause, diagnosis commands (copy-pastable, with placeholders marked), mitigation, and the point at which to escalate.
4. Safe actions: restart, roll back to the fallback version, disable a feature toggle, scale up, with their side effects stated.
5. Dangerous actions: what not to do without a second person (schema changes, data deletion, credential rotation).
6. Escalation: who to contact in which order and what to tell them.
7. Test the runbook by having a colleague follow it during a game day; fix every step they stumbled on; link it from each alert.

Expected result

Incidents are handled by the person on call rather than by waking the author; postmortems produce runbook updates.

Limits and test basis

Runbooks describe known failures; novel ones still need understanding. Untested runbooks are worse than none because they inspire false confidence. No measurement is claimed.