Notification service walk-through: channels, preferences, delivery attempts and retries

Goal

Deliver messages from many producers to users over several channels according to their preferences, so that no producer talks to a provider directly and a retry never sends twice.

Prerequisites

A list of channels (email, push, SMS, in-app) and their providers, a catalogue of notification types with a default channel and urgency per type, and a policy on what a user may switch off.

Steps

1. Constraints: producers must not block on delivery; providers fail independently; for most types a duplicate is worse than a delay; bulk campaigns must not delay transactional messages.
2. Components: an ingest API that accepts (idempotency_key, recipient, type, payload) and writes a row before answering; a router that expands one notification into per-channel deliveries after checking preferences and quiet hours; one queue per channel; channel workers that render templates and call providers; a callback receiver for bounces, complaints and invalid tokens.
3. Data model: notification(id, idempotency_key unique, recipient, type, payload, created_at); delivery(id, notification_id, channel, address, status, attempts, next_attempt_at, provider_message_id, last_error); preference(recipient, type, channel, enabled); device_token(recipient, token, platform, invalidated_at). The unique key makes producer retries harmless.
4. Retries: each channel has its own backoff schedule with jitter and a maximum; classify provider errors as retryable (timeouts, 5xx, rate limits) or terminal (invalid address, unsubscribed). For Web Push, RFC 8030 requires a TTL header on every delivery request, in seconds, and states that once it elapses the push service must not attempt delivery; an urgent notification should carry a short TTL rather than be retried for hours after it stopped being useful.
5. Failure modes: a worker crashing after the provider accepted but before the row was updated (a duplicate; keep the provider message id, prefer providers with idempotent send APIs); provider outage growing the backlog (drop time-sensitive types by age instead of delivering them late); a template failing for one locale (fail that delivery, not the worker); campaigns starving transactional traffic (separate queues or priorities); tokens invalidated silently (process callbacks and prune).
6. Measure: time from ingest to provider acceptance per channel, attempts per delivery, terminal failure rate by error class, queue age, opt-out rate per type.
7. Not first: multi-provider failover, digests and batching, per-user rate caps, a template editor, engagement analytics, copy experiments.

Expected result

Producers call one API; each delivery is either acknowledged by a provider, terminally failed with a reason, or still scheduled; replaying a producer request produces no second message.

Limits and test basis

Proposed design, no measurements. Duplicate suppression at the provider boundary depends on the provider; without an idempotent send API the design accepts rare duplicates after a crash.