Logs, metrics and traces: which signal answers which question

What it is

Observability rests on three types of signal, which OpenTelemetry defines in a unified way. A log is a timestamped record of an event with arbitrary fields; among other things, the OpenTelemetry data model provides the fields TraceId and SpanId so that a log entry can be matched to the request in which it was produced. A metric is a numeric time series; the documentation distinguishes instruments such as Counter (only increases, like an odometer), UpDownCounter (increases and decreases, for example a queue length), Gauge (the current value at the time of reading) and Histogram (a distribution, for example of response times). A trace represents a request as a tree of spans, each with a start, a duration, attributes and a parent span, and context propagation carries the trace ID to downstream services via headers.

Why it matters

Each signal answers a different question, and none replaces the others. Metrics are cheap to store and query and are well suited for alerts: "Error rate above two percent for five minutes." However, they do not tell you which request was affected. For a given request, traces show where the time went: the database, an external service, or a serial call that could run in parallel. Logs contain the exact wording of the error and the values that led to it. Only the shared identifier turns three tools into one investigation: from the alert to a sample request to the log entry.

How to apply

• Start with the question, not the tool: for "is the service healthy?" a few metrics are enough (requests, errors, duration, saturation); for "why is this request slow?" you need traces; for "what was in the input?" you need logs.
• Instrument the boundaries first: incoming HTTP requests, outgoing calls, database access. Libraries usually provide this automatically.
• Limit metric labels to a few values (endpoint, status class), and never use user or object identifiers: every combination is its own time series.
• Write the trace ID into every log record and pass it on to downstream services; without propagation, the trace ends at the first service boundary.
• Sample traces when volume is high: if the decision is made at the start, sampling is cheap; if it is made after the trace is complete, failed and slow requests can be kept in full.
• Set retention per signal: metrics long and aggregated, logs shorter, traces shortest.

Pitfalls

High-frequency logs used as a substitute for metrics ("one line per request, then we count") do not scale and cost money on every query. A histogram that exports only averages loses the outliers that users actually notice. Traces without attributes (status code, row count, cache hit) answer only the "where", not the "why". Collecting everything without having a question produces costs, not insight.