Which trace sampling strategy keeps rare failures visible in a low-traffic service?

Open question

Published sampling advice tends to assume volume: keep a small percentage, sample at the tail to retain errors and slow requests, and rely on the law of large numbers for the rest. The OpenTelemetry sampling page itself lists generating very little data (tens of small traces per second or lower) among the reasons not to sample at all, and stops there. A small service sees a few requests per second, most of them healthy, and the events that matter are rare: one timeout a day, one malformed payload a week. A fixed head-sampling rate discards most of those; keeping everything is often affordable at this volume but retention then becomes the cost lever, and tail sampling needs a collector that buffers whole traces.

What have operators of low-traffic services actually done, and what did it cost them? Candidates include keeping every trace for a short window and only error or slow traces for a long one; head sampling at 100 percent with a per-route override for health checks and crawlers; tail sampling with rules for status, latency and specific attributes; and dynamic rates that rise when traffic falls. It is unclear which of these teams sustained, which they abandoned, and how often an incident review found that the needed trace had been sampled away.

What a useful answer contains

The service's request rate and the number of services per trace; the sampling mechanism and its rules; retention per class of trace; the storage and collector cost in a stated unit; examples of incidents where the trace was present and where it was missing; and whether the configuration survived a change in traffic or team. Vendor defaults should be named as such, and anecdotes labelled as single cases.