Designing alerts well: few notifications, each with a next step

Goal

An alert set in which every notification triggers an action and nobody learns to ignore the pager.

Prerequisites

The chapter "Monitoring Distributed Systems" in the Google SRE book names the four golden signals (latency, traffic, errors and saturation), distinguishes symptoms ("what is broken") from causes ("why"), and requires that every page be actionable ("Every page should be actionable"). The SRE Workbook describes alerting based on the error budget consumption rate ("burn rate"). In practice you need measured service indicators at the user edge, an alerting system with a wait time before firing (in Prometheus the for clause, supplemented by keep_firing_for against flapping while an alert is resolving), and a documented escalation path.

Steps

1. For each service, define the symptoms that users notice: error rate, latency percentiles, availability, data freshness. Only these may wake someone. Causes such as high CPU load or a disk that is 80 percent full become tickets or dashboard tiles.
2. Derive thresholds from the objectives: the Workbook gives, as a starting point for a page, consuming 2 percent of the error budget in one hour or 5 percent in six hours; a slower rate (10 percent in three days) creates a ticket instead of a page.
3. Give every rule a wait time (for: 10m) so that a single outlier does not fire it, and add a rule that fires when the monitoring system itself stops delivering data.
4. In the annotations of every alert, include a runbook link, the affected dashboards and the first diagnostic step.
5. Carry urgency as a label (severity: page versus ticket) and route notifications accordingly; at night only the first tier reaches a person.
6. Every week, review all alerts that fired: every alert that led to no action is retuned or deleted; every incident that had no preceding alert gets one.

Expected result

Few pages, each with a clear next step; incidents are noticed by monitoring before users notice them; the on-call team trusts the pager.

Limits and test basis

Symptom alerts detect gradual degradation late; a few leading indicators with low urgency (queue length, a forecast such as "disk full in two days") complement them. The principles follow the cited SRE chapters and the mechanics follow the cited Prometheus documentation; whether a team becomes desensitized, and at what alert rate, is not substantiated here.