Publishing events reliably with a transactional outbox

Goal

Guarantee that every committed state change produces its event exactly when the change is committed, without a distributed transaction between the database and the message broker.

Prerequisites

A service that owns its database and publishes events to a broker; consumers that tolerate duplicates (deduplicate by message id). The pattern page on microservices.io (cited) describes the problem: a service must atomically update its database and send a message, and messages for one aggregate must keep their order across service instances.

Steps

1. Create an outbox table: id (unique, becomes the message id), aggregate_type, aggregate_id, event_type, payload (JSON), created_at, and, for the polling variant only, published_at (nullable). Debezium's default column names are id, aggregatetype, aggregateid, type and payload; other names are mapped through its options.
2. In the application transaction that changes state, insert one outbox row per event. Commit. Nothing else happens in the request path.
3. Choose a relay:
   • Polling publisher: a worker selects unpublished rows in id order (FOR UPDATE SKIP LOCKED in PostgreSQL to allow several workers), publishes each to the broker with the row id as message id and the aggregate id as partition key, then sets published_at.
   • Log tailing: a change-data-capture connector reads the database log. Debezium's outbox event router (cited) captures inserts into the outbox table, routes each row to a topic derived from the aggregate type and uses the aggregate id as the message key. Its documentation states that updates to outbox rows are not allowed and that deletes are filtered out, so with this variant the table is insert-only: rows are deleted after the fact, never marked.
4. Accept that a crash between publishing and marking (or, with log tailing, between publishing and the connector recording its position) produces a duplicate; broker-side producer idempotence, where offered, covers retries within one producer session, not a restarted relay. Consumers deduplicate by message id.
5. Delete or archive published rows on a schedule; keep the table small so the poll query stays cheap.
6. Monitor the age of the oldest unpublished row and the count; alert when the relay stalls.

Expected result

No event without a committed change and no change without an event. Consumers see each event at least once, in per-aggregate order if the relay preserves insertion order and the broker preserves order per key.

Limits and test basis

Polling adds latency of one poll interval; log tailing needs CDC infrastructure and database permissions. Order across different aggregates is not guaranteed. Test by killing the relay mid-batch and by crashing the application between the business write and commit; the outbox must show neither orphaned events nor missing ones.

Ordering with a polling relay

Several polling workers using SKIP LOCKED do not preserve per-aggregate order: one worker can publish a later row for an aggregate before another worker publishes an earlier one, and a row with a lower id can become visible after a row with a higher one because transactions commit out of id order. If consumers depend on per-aggregate order, run a single publishing worker, or partition workers by a hash of aggregate_id so that one aggregate's rows are always handled by the same worker, and poll from the oldest unpublished row rather than from the last id seen. The log-tailing relay avoids both problems because it reads commits in commit order.