Alert hygiene for host monitoring: rate over threshold, failed units, and duration windows

What it is

Google's SRE book states that "paging a human is a quite expensive use of an employee's time" and that effective alerting needs "good signal and very low noise" — every page that turns out to need no action trains the on-call engineer to trust pages less. The SRE workbook's chapter on alerting on SLOs describes multiwindow alerting, which fires only when the burn rate is high over both a long window (so a brief spike does not page) and a short one (so the alert clears soon after recovery).

Applied to a single host rather than a service's SLO, the same two ideas answer which of the signals covered elsewhere in this series belong on a pager versus a ticket queue:

• Disk space: a fixed percentage (say, "page at 90% full") pages identically whether the partition has been sitting at 91% unchanged for six months or filled from 60% to 91% in the last hour. The fill rate — projected time to full at the current rate of growth — is the signal that actually predicts an imminent outage; a slow, stable 91% is a ticket, a partition on track to fill within the hour is a page.
• Failed systemd units / stopped Windows services: a unit that is failed right now and expected to be running is close to the "already broken" case the SRE book treats as page-worthy; a unit that failed once and was already restarted by its own retry logic is a ticket, not a page.
• Clock sync: a host whose clock has drifted enough to break TLS validation or log correlation is page-worthy; a momentary NTP query timeout that resolves on the next poll is not.
• SMART/NVMe health: a failed self-test or a nonzero critical-warning byte is page-worthy given the short window before data loss; a single reallocated sector appearing on an otherwise healthy drive is a ticket to watch.

Why it matters

Alerting on the raw instantaneous value of a metric, with no duration or trend requirement, is what produces the flapping and noise the SRE book warns against: a threshold hovering near its boundary fires and clears repeatedly, and the resulting alert fatigue is the same failure mode whether the metric is a disk percentage or an HTTP error rate.

How to apply

• For every host alert, ask whether the rule reacts to a rate/trend or a raw level, and whether it requires the condition to persist across a window rather than an instant sample.
• Route "already broken, action needed now" conditions (disk about to fill within the on-call window, a required unit down, an unsynced clock) to a page; route "worth reviewing soon" conditions (slow, stable, or already-recovered) to a ticket or dashboard.
• Re-evaluate any alert that pages more than a handful of times without a corresponding real fix — the SRE book's Bigtable case study describes disabling email alerts that had become too numerous to diagnose.

Pitfalls

• Copying a percentage-based disk threshold from one host to another with a very different growth rate, where the same number means a different amount of runway.
• Treating a page and a ticket as the same severity with different delivery channels, rather than as different classes of urgency.