Alert routing: grouping, inhibition, silences and escalation policies

Goal

Deliver each alert to the one person or channel that can act on it, as one notification per incident rather than one per instance, with a defined path when nobody answers.

Prerequisites

Alerts that carry labels for severity, team and service; a routing component (the Alertmanager documentation describes deduplicating, grouping, routing to receivers, silencing and inhibition); and a paging service with escalation policies.

Steps

1. Fix the label contract first: severity (page, ticket), team, service. Routing matches on labels, so an alert without them lands in the default receiver and is nobody's.
2. Build the routing tree: a root receiver for the unmatched remainder, child routes per team, and within them a page receiver for severity=page and a ticket or chat receiver for the rest. Use continue: true only where an alert must reach two receivers on purpose.
3. Group by alertname and service (or cluster) so that an outage with fifty instances produces one notification listing fifty. Set group_wait near the documented example (30s): too short, the documentation warns, sends an incomplete first notification and lets through alerts that a late-arriving source alert would have inhibited; too long delays the first notification. Set group_interval for updates to an existing group and repeat_interval (the documented example uses 4h) for re-sending an unresolved group.
4. Write inhibition rules for known cause-and-symptom pairs: when the cluster-unreachable alert fires, mute every alert about that cluster; the source alert must carry the same value in the matching labels.
5. Use silences, with an expiry and a comment, for planned maintenance, and time intervals on routes for recurring quiet hours of ticket-level alerts; never silence a page-level alert without a ticket that says why.
6. Define the escalation policy in the paging service: first rule notifies the on-call person, an escalation timeout gives them a fixed number of minutes to acknowledge, resolve or reassign, then the next rule notifies the secondary, optionally repeating the policy. Acknowledging stops escalation; only resolving ends the incident.
7. Test end to end with a synthetic alert per route each quarter and after every change to the tree: right receiver, one notification, escalation reached the secondary when the primary ignored it.

Expected result

A page names one service and lists its affected instances, arrives at the responsible on-call person within the group wait, and reaches a second person automatically if the first does not acknowledge.

Limits and test basis

Timings here are documented defaults and examples, not tuned values; each team sets its own from the escalation timeout backwards. Inhibition only works when labels are consistent across alert sources. A routing tree is configuration that drifts: it needs the same review as code.