Latency percentiles: why the average describes no real request

What it is

Request latency is not distributed symmetrically: most requests are fast and a few are very slow, with no upper bound short of the timeout. The arithmetic mean of such a distribution lies between the fast majority and the slow tail and describes no actual request. Percentiles describe the distribution directly: p50 (the median) is the typical request, p99 is the value that one request in a hundred exceeds, and the maximum is the worst request in the window. The SRE book gives the example of a service with an average latency of 100 ms at 1,000 requests per second where 1% of requests might easily take 5 seconds, and adds that the 99th percentile of one backend can easily become the median response of a frontend that depends on several such backends.

Why it matters

A user who makes 100 requests during a session meets the p99 at least once with probability 1 − 0.99^100, about 63% (arithmetic). Alerts on the mean fire late or never, service level objectives are written in percentiles, and a "10% faster on average" change can leave the tail untouched or make it worse.

How to apply

• Instrument with histograms: counts of requests per latency bucket. The SRE book suggests bucket boundaries spaced roughly exponentially. The Prometheus documentation explains that quantiles pre-computed in the instrumented program cannot be aggregated across instances and cannot be recomputed for another window or percentile, whereas histograms can; it also notes that quantiles derived from histograms are estimates whose error depends on bucket width around the value of interest.
• Report p50, p90, p99 and max together with the request count. A p99 over 100 requests is a single request; state the sample size.
• Choose the percentile from exposure: an endpoint hit once per page view can be judged at p90; a call made fifty times per page needs its p99 or p99.9.
• Keep timeouts and errors as separate series. A timeout caps the measured latency and would otherwise hide the true tail.
• Measure at the client as well as the server. Time spent in a connection backlog or load balancer queue is invisible to the server's own histogram.

Pitfalls

Averaging p99 values across hosts or minutes produces a number that is neither an average nor a percentile; aggregate the histograms and recompute. Percentiles over tiny samples are noise. A closed-model load generator that waits for slow responses under-samples the slow periods, so its percentiles are optimistic (see the article on open and closed workload models). Bucket boundaries that stop at 1 s make every slower request look like 1 s.