Log sampling for high-volume events: keep every error, sample the repetitive lines

What it is

Log sampling records only a chosen fraction of events that would otherwise repeat thousands of times with little variation: per-request debug lines, cache hits, health-check requests, retries in a loop. The zerolog README shows the common producer-side shapes: a basic sampler that keeps one event in N, a burst sampler that lets a number of events through per period and then hands the rest to a one-in-N sampler, and a level sampler that applies these rules to debug events only while other levels are not sampled. Pipeline-side tools do the same after the fact; Vector's sample transform keeps a configurable rate (one in N) or ratio, can hash a key_field so that all events sharing a value are kept or dropped together, records the applied rate in a field, and takes an exclude condition for events that must never be sampled. The OpenTelemetry sampling page describes the trace-side counterpart: head sampling decides early and cheaply, tail sampling decides after seeing the whole trace and can therefore keep every trace with an error, which head sampling alone cannot guarantee.

Why it matters

Log volume is paid three times: CPU and I/O on the producer, bandwidth and indexing in the pipeline, and storage for the retention period. Sampling cuts all three, but only if what survives still answers the questions asked during an incident.

How to apply

• Never sample warnings, errors, audit events or security events; apply sampling to informational and debug levels only.
• Key the sampling on the request or trace ID so a kept request is complete and a dropped one is absent, rather than every request being half-logged.
• Write the sampling rate into the kept events (sample_rate: 100) so counts can be scaled back and readers know what they are looking at.
• Move counting to metrics first; a counter costs one sample per scrape however many events it counted, while logging costs one line per event.
• Prefer burst-then-rate over plain one-in-N for events that matter when they start happening: the first occurrences of a new failure mode arrive unsampled.
• Keep a switch to disable sampling for one service during an investigation, and remember to turn it back on.

Pitfalls

Hashing on a key whose values are unevenly distributed changes the effective rate, as the Vector documentation notes. Alerts or dashboards that count log lines silently break when sampling starts. Sampling in the producer and again in the pipeline compounds the rates. Compliance logs that must be complete are not a cost problem to solve with sampling.