Synthetic monitoring and uptime checks: probing from outside what users see

Goal

Detect that users cannot reach or use the service, independently of whether the service's own metrics, logs or health endpoints are working, and record availability from the outside.

Prerequisites

A probing tool that runs outside the production network (the Prometheus blackbox exporter probes HTTP, HTTPS, DNS, TCP, ICMP and gRPC targets and exposes probe_success plus timing metrics), at least two probe locations, and an alerting path that does not depend on the monitored system. The SRE book's distinction applies: black-box monitoring is symptom-oriented and reports active problems ("the system is not working correctly, right now"), while white-box monitoring inspects internals and can see imminent problems.

Steps

1. List what a user needs in sequence: DNS answer, TLS handshake, the landing page, the login or API entry point, one read that touches the database, one static asset from the CDN.
2. Write one probe per step with a correctness condition, not only a status code: expected body substring or JSON field, expected redirect target, expected DNS record value, certificate validity.
3. Set the probe timeout below the probe interval; the blackbox exporter README notes that a Prometheus scrape timeout can never exceed the scrape interval.
4. Run each probe from at least two locations on different networks; treat a failure as real only when it persists for several consecutive intervals at more than one location.
5. Alert on probe_success == 0 under that rule and keep the duration series for latency trends; sudden slowness from one location usually means a network path, from all locations the service.
6. Mark probe traffic (a dedicated user agent or header) so it is excluded from analytics, rate limits and security alerts.
7. Rehearse: point a probe at a deliberately broken staging target and confirm the page arrives through the external path.

Expected result

An availability record from the user's side that pages within a few intervals of an outage and that keeps working when the internal monitoring stack is the thing that failed.

Limits and test basis

A probe sees one path with one client; it cannot see degradations that return a correct page slowly for some users, nor logged-in journeys unless a test account and idempotent actions exist. Probing write endpoints in production needs such accounts and cleanup. Probe intervals and location counts here are design choices, not measured optima.