Sending transactional email reliably: outbox row, worker, retries and idempotency keys

Goal

Every order confirmation, password reset or invoice leaves the system once, survives process crashes and provider outages, and can be traced from the business event to the provider's message identifier.

Prerequisites

A durable store (the application's own database is enough), a worker process, a provider or MTA reachable over SMTP or HTTP, and a Message-ID convention. The cited RFC 5321 splits negative replies into transient (4yz: the same request may succeed later) and permanent (5yz: do not repeat the request unchanged) classes; the cited RFC 5322 requires the Message-ID to be a globally unique identifier.

Steps

1. In the same database transaction that records the business event, insert an outgoing_email row: recipient, template name, rendered parameters, an idempotency key derived from the event (order-1234-confirmation), a freshly generated Message-ID and status pending. This is the outbox pattern applied to mail.
2. A worker claims a batch of pending rows (in PostgreSQL: SELECT ... FOR UPDATE SKIP LOCKED), renders the message and calls the provider with the stored Message-ID and, where the API accepts one, the idempotency key.
3. Classify the outcome. 4yz replies, connection errors and timeouts are retryable: keep the row, increment attempts, set next_attempt_at with exponential backoff and jitter. 5yz replies and rejected addresses are final: mark failed with the reply text and stop.
4. Cap attempts and age. RFC 5321 says an MTA's give-up time generally needs to be at least 4-5 days; an application queue usually caps much earlier, and rows beyond the cap move to a dead-letter status that raises an alert.
5. Store the provider's message identifier and timestamps on the row so bounces, complaints and support questions can be joined back to the event.
6. Make the worker safe to run in parallel and after crashes: the unique idempotency key or Message-ID is what prevents a second send when the process dies between "provider accepted" and "row marked sent".
7. Use separate queues or priorities for time-critical mail (login codes, resets) and everything else, so a bulk burst cannot delay a code.

Expected result

Sends survive restarts; a provider outage shows up as a growing pending count rather than as lost mail; duplicates after a crash are prevented by the key rather than by luck.

Limits and test basis

"Exactly once" holds only up to the provider's API: a provider that accepted the request but never answered may still have sent, and provider-side idempotency keys are typically honoured only for a period the provider documents. Test by killing the worker between send and mark-sent, and by simulating 4yz and 5yz replies. Based on the cited RFCs and documented practice; no timings or rates are claimed.