SLIs for queues and batch jobs: age of the oldest message, freshness, coverage and last success

Goal

Define service level indicators for asynchronous work so that "the queue is fine" and "the nightly job ran" become measured statements with a threshold.

Prerequisites

A list of the queues and scheduled jobs, the expectation users have of each (how stale may the result be, by when must the run finish), and a metrics system storing gauges and ratios. The SRE workbook's component types (request-driven, pipeline, storage) are the vocabulary; its pipeline SLIs are freshness, correctness and coverage.

Steps

1. Classify each component. A consumed queue and a scheduled job are both pipelines in the workbook's sense: records go in, results come out later.
2. For a queue, write the user-facing indicator first: the proportion of items processed within N seconds of enqueueing (freshness). The practical proxy is the age of the oldest unprocessed item; hosted queues expose it (Amazon SQS reports ApproximateAgeOfOldestMessage in seconds). Backlog size is a cause indicator, useful for capacity, not the SLI.
3. For a batch job, export the timestamp of the last successful completion as a gauge; the Prometheus instrumentation guide calls this the key metric of a batch job and recommends pushing it, with stage durations and records processed, because a job that does not run continuously is hard to scrape.
4. Add coverage: for batch processing, the proportion of runs that processed at least the expected amount of data; for streaming, the proportion of incoming records processed within the window. A run that finishes instantly because its input was empty is a coverage failure, not a success.
5. Add correctness where a checker exists: the proportion of input records whose output is right, measured on a sample against a reference computation.
6. Turn each indicator into a ratio over a window (good events divided by total events), set a target, and write the alert as "time since last success exceeds twice the schedule period" or "oldest item older than the freshness threshold for M consecutive evaluations".
7. Record indicator, implementation, target and window in the SLO document.

Expected result

Each queue and job has a freshness or last-success indicator with a threshold, a coverage check that catches empty runs, and an alert that fires on absent data as well as bad data.

Limits and test basis

Hosted-queue age metrics are documented as approximate; the SQS guide notes that a standard-queue message received three or more times without deletion moves to the back of the queue and leaves the age metric, so a poison message does not appear as growing age. A job that never starts emits nothing, so alerts must treat a missing series as failure. Correctness needs an independent reference and is usually sampled.

Deadline-based alerts for scheduled jobs

The 'twice the schedule period' rule suits jobs whose only requirement is regularity. A job with a consumer deadline (a report due at 06:00 from a 03:00 run) needs an alert derived from the freshness indicator instead: fire when the deadline has passed and the last success is older than the scheduled start, for example hour() >= 6 and time() - job_last_success_timestamp_seconds > 3 * 3600, evaluated in the hours after the run and expressed in the timezone the schedule uses. Write the deadline next to the schedule in the SLO document, keep the period-based rule as the coarser fallback for jobs without a stated consumer, and make both rules treat a missing series as a failure with absent().