Metric naming and label cardinality: units in the name, bounded values in the labels

What it is

A metric name says what is measured; labels say for which slice. The Prometheus naming guide gives the conventions: a single-word application or subsystem prefix (http_, process_), one unit and one quantity per metric, base units (seconds, bytes, not milliseconds or megabytes), the unit as a plural suffix, _total for accumulating counts and _info for metadata pseudo-metrics. Examples from the guide: http_request_duration_seconds, process_cpu_seconds_total, http_requests_total. The guide also states a test: the sum() or avg() over all label dimensions of one metric should be meaningful; if it is not, the data belongs in two metrics. It notes that other conventions, OpenTelemetry among them, keep the unit out of the name and in metadata; Prometheus recommends keeping it in the name so that alert and dashboard configuration remains readable without a schema.

Labels are the dimensions: operation="create", stage="extract", a status code, a route template. The guide's caution is the important half: every unique combination of label values is a new time series, so labels must not carry unbounded sets such as user IDs or email addresses.

Why it matters

A series costs memory, CPU, disk and network for as long as it exists. The instrumentation guide gives orders of magnitude: keep the cardinality of a metric below 10 as a guideline, treat anything over 100 or with the potential to grow as a case for fewer dimensions or for moving the analysis out of the monitoring system, and expect the vast majority of metrics to have no labels at all. A classic histogram ingested the classic way multiplies the problem: each configured bucket creates a series suffixed _bucket for every label combination, whether populated or not.

How to apply

• Name by prefix, quantity, unit, _total: queue_consumed_messages_total, job_last_success_timestamp_seconds.
• Use route templates, not raw paths; status classes or codes, not messages; enum-like values only.
• Before adding a label, estimate its distinct values and multiply by the existing label combinations and by histogram buckets.
• Move per-user or per-request analysis to logs or traces, where high cardinality is normal.
• Set a per-target sample limit where the server offers one (Prometheus: sample_limit in the scrape configuration) and alert on series count growth so a bad label is caught before the server is.

Pitfalls

Putting a label's meaning into the metric name (http_requests_get_total) breaks aggregation. Mixing seconds and milliseconds under one name makes the series collide silently. Pod or container names on application metrics are bounded but churn on every deploy, creating a new set of series each time. Native histograms change the bucket cost model; the cited histogram guide explains the variants.

What a sample limit does when it trips

sample_limit is a last line, not a filter: when a target exceeds it, Prometheus treats the whole scrape as failed, so every metric of that service disappears at once and the alerts that read them go silent. Use it in three steps. Drop or aggregate the known-risky labels in metric_relabel_configs, so a bad label costs one metric rather than the target. Alert on scrape_samples_post_metric_relabeling rising towards the limit per target, and on the server's prometheus_tsdb_head_series, so growth is seen before the cut. Then set sample_limit well above the normal count so that only a runaway trips it. label_limit and the label length limits fail the scrape in the same way and deserve the same margin.