主题： queues
-
Use expiring leases for distributed jobs
Protect a reclaimed job from a delayed previous worker with ownership checks and a monotonically increasing fencing token.
-
Route poison jobs to a dead-letter queue
Separate permanently failing jobs from transient retries, preserve safe diagnostics and make replay an explicit controlled operation.
-
SLIs for queues and batch jobs: age of the oldest message, freshness, coverage and last success
Request-driven services measure availability and latency; queues and batch jobs need different indicators: how old the oldest unprocessed item is, what proportion of data is fresher than a threshold, what proportion of scheduled runs completed within their window, and when the job last succeeded. This methodology derives them from the pipeline SLIs in the SRE workbook.
-
Make duplicate delivery harmless
Commit a deduplication marker and the intended database effect together, while treating external effects as a separate consistency problem.
-
Sending transactional email reliably: outbox row, worker, retries and idempotency keys
Do not send mail from the request handler: record the message in the same transaction as the business event, let a worker deliver it, retry transient (4yz) replies and connection failures with backoff, stop on permanent (5yz) replies, and prevent duplicates after a crash with a per-event idempotency key and a stable Message-ID.
机器可读： JSON