The RED method for request-driven services, with exemplars that link a slow bucket to a trace

Goal

Give every request-driven service the same three signals so that dashboards, alerts and incident questions look alike across the architecture: Rate (requests per second), Errors (requests that fail) and Duration (how long requests take), the RED method, which the cited Grafana post attributes to Tom Wilkie as a microservices-oriented counterpart to the USE method, which applies to hardware resources.

Prerequisites

A metrics library with counters and histograms, low-cardinality route templates, and optionally a tracing setup whose trace IDs can be attached to metric observations.

Steps

1. For each service, expose one counter <svc>_requests_total{route, method, code} incremented when a request ends, and one histogram <svc>_request_duration_seconds with the same labels. Errors are derived from code (5xx, or a domain-specific failure label) rather than a separate counter that can drift from the total.
2. Build one dashboard row per service: request rate, error ratio (errors divided by total over the same window), and duration percentiles from the histogram. Order rows by data flow so the row above is the caller of the row below.
3. Alert on the error ratio and on the duration objective, not on the rate; rate changes are context.
4. Attach exemplars. OpenMetrics defines an exemplar as a reference to data outside the metric set, most commonly a trace ID, consisting of a label set and a value with an optional timestamp, the combined length of label names and values limited to 128 code points. The OpenTelemetry metrics SDK specification describes an exemplar as a recorded measurement that exposes the value, the time of the call, attributes dropped by aggregation and, for synchronous instruments, the trace ID and span ID of the active span; its default exemplar filter should be TraceBased, so only measurements made inside a sampled span are eligible to become exemplars. Other client libraries may need the trace ID passed with the observation; check the library's documentation.
5. Enable exemplar storage in the metrics backend and configure the dashboard's histogram panels to show exemplar points that open the trace on click.
6. Check the join: pick a slow bucket on the dashboard, open the exemplar, and confirm the trace shows the request that produced the observation.

Expected result

Any responder can read any service's health in the same three panels, and can go from "p99 rose at 14:02" to one concrete trace of a slow request without searching.

Limits and test basis

RED covers request-driven components; queues and batch jobs need freshness and last-success indicators, and resource exhaustion needs USE. Exemplars are only as good as trace sampling: an unsampled slow request leaves no exemplar. Bucket boundaries decide which percentiles a classic histogram can answer.