Outbox Pattern Internals: Ordering Guarantees, Relay Mechanics, and the Failure Modes Nobody Documents

The outbox pattern solves one hard problem—atomic pairing of a database write with a downstream event emission—but it introduces a different set of hard problems that most write-ups skip entirely. Correctness at the business transaction boundary is only the beginning. Everything after that involves tradeoffs that compound under real production conditions: relay scheduling, ordering semantics across partitions, duplicate delivery windows, and what happens when your relay process restarts mid-batch.

This article examines those mechanics, with Go examples where they clarify the design.

The Core Guarantee and Its Exact Scope

The outbox pattern gives you this and only this: a business state change and the intent to emit an event are committed atomically to the same database transaction. If the transaction commits, both exist. If it rolls back, neither does. You eliminate the dual-write race where a service writes to the DB, crashes, and the broker never receives the event—or worse, the broker receives the event but the DB write never lands.

What it does not give you:

• At-most-once delivery
• Strict global ordering across consumers
• Low-latency emission (the relay adds a processing hop)
• Guaranteed ordering between events from different transactions, even within the same aggregate, unless you design for it explicitly

Understanding the boundary of the guarantee is what separates a correct implementation from one that works until load or failure exposes the assumptions.

Relay Implementation: Polling vs. WAL Tailing

Two viable approaches exist for the relay: polling and WAL-based change data capture (CDC).

Polling Relay

A relay goroutine periodically queries the outbox table for unprocessed rows, publishes them to the broker, then marks them delivered.

func (r *OutboxRelay) Run(ctx context.Context) error {
    ticker := time.NewTicker(r.pollInterval)
    defer ticker.Stop()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-ticker.C:
            if err := r.processBatch(ctx); err != nil {
                r.metrics.RelayErrors.Inc()
                r.log.Error("relay batch failed", zap.Error(err))
            }
        }
    }
}

func (r *OutboxRelay) processBatch(ctx context.Context) error {
    rows, err := r.store.FetchUnprocessed(ctx, r.batchSize)
    if err != nil {
        return fmt.Errorf("fetch: %w", err)
    }
    for _, row := range rows {
        if err := r.publisher.Publish(ctx, row); err != nil {
            return fmt.Errorf("publish row %s: %w", row.ID, err)
        }
        if err := r.store.MarkDelivered(ctx, row.ID); err != nil {
            return fmt.Errorf("mark delivered %s: %w", row.ID, err)
        }
    }
    return nil
}

Critical failure mode: if Publish succeeds but MarkDelivered fails, the next poll cycle republishes the same row. This means your downstream consumers must handle duplicates—idempotency on the consumer side is not optional, it is load-bearing. This is at-least-once delivery by construction.

A second failure mode: the FetchUnprocessed query does a full or partial table scan unless you maintain a covering index on (status, created_at). Under write-heavy load, the outbox table grows faster than the relay drains it. Monitor outbox_unprocessed_count and alert before this becomes a multi-minute lag.

WAL Tailing (CDC)

Tools like Debezium or a custom Postgres logical replication client subscribe to the WAL stream. The relay processes INSERT events on the outbox table directly from the replication slot, without polling.

Advantages: sub-second latency from commit to relay, no polling load on the primary, and the replication slot's LSN provides a durable cursor—restart the relay and it picks up exactly where it left off without re-scanning.

Disadvantages: operational complexity (replication slots must be monitored—unconsumed slots cause WAL retention to grow unboundedly), and the relay is coupled to a specific database's replication protocol. If you're running MongoDB, you'd use the change stream equivalent; the mechanics differ but the ordering properties are the same.

Operational rule: set max_slot_wal_keep_size in Postgres and alert on replication slot lag separately from relay message lag. They can diverge.

Ordering: What You Actually Get

Ordering is where most outbox implementations make implicit assumptions that fail under concurrency.

Consider two concurrent transactions on the same aggregate (say, order_id = 42):

• Tx A commits at T=100ms, inserts outbox row with id=1
• Tx B commits at T=101ms, inserts outbox row with id=2

If your relay fetches by insertion order and processes sequentially, you get ordered delivery for this aggregate. But that's a best-case scenario.

Failure mode—relay restart gap: Tx A commits. The relay fetches row id=1, publishes it, then crashes before marking it delivered. Tx B has also committed; its row id=2 is now also unprocessed. On restart, the relay fetches both. Depending on your FetchUnprocessed query and sort order, id=2 may be processed before id=1 is re-confirmed as delivered. You now have a window where downstream consumers receive events out of insertion order for the same aggregate.

The robust mitigation is to include a sequence number scoped to the aggregate in the outbox row and enforce ordering on the consumer side, not by trusting relay delivery order.

type OutboxRow struct {
    ID          string
    AggregateID string
    Sequence    int64  // monotonic per aggregate, set in the same transaction
    EventType   string
    Payload     []byte
    Status      string
    CreatedAt   time.Time
}

Consumers that need strict per-aggregate ordering must buffer and reorder by (AggregateID, Sequence) before processing. Cross-aggregate ordering—event from order 42 before event from shipment 99—is generally not achievable with an outbox unless you introduce a distributed sequence, which is usually not worth the coordination cost.

Duplicate Suppression on the Consumer Side

Because at-least-once is the delivery semantic, consumer idempotency must be explicit and durable. Memoizing in memory is not sufficient—relay restarts after a crash will replay.

A practical pattern: maintain a processed_events table (or Redis set with a TTL longer than your maximum replay window) keyed by event_id. Wrap the business logic and the idempotency record insertion in a transaction:

func (h *OrderHandler) Handle(ctx context.Context, evt Event) error {
    return h.db.WithTransaction(ctx, func(tx *sql.Tx) error {
        var exists bool
        err := tx.QueryRowContext(ctx,
            `SELECT EXISTS(SELECT 1 FROM processed_events WHERE event_id = $1)`,
            evt.ID,
        ).Scan(&exists)
        if err != nil {
            return err
        }
        if exists {
            return nil // idempotent skip
        }
        if err := applyBusinessLogic(ctx, tx, evt); err != nil {
            return err
        }
        _, err = tx.ExecContext(ctx,
            `INSERT INTO processed_events(event_id, processed_at) VALUES($1, NOW())`,
            evt.ID,
        )
        return err
    })
}

The processed_events table needs a cleanup job—events older than your guaranteed replay window can be pruned. Without pruning, it becomes a performance liability. Index on event_id; if volume is high, partition by month.

Outbox Table Schema and Retention

The outbox table is a write-amplification surface. Every business transaction writes at least one outbox row in addition to the business entity row. Under high throughput this matters.

Schema considerations:

• status should be a narrow column (pending/delivered) with a partial index on status = 'pending' for relay fetch performance
• payload as bytea or jsonb—jsonb adds indexing capability but parse overhead; bytea is faster to scan when you don't filter on payload content
• Add retry_count and last_error for relay diagnostics without needing separate error storage
• Archive or delete delivered rows on a schedule; do not let the table grow unboundedly

Concurrency in the Relay: Why You Almost Never Want Multiple Relay Workers on the Same Queue

Running two relay instances for throughput seems straightforward. It isn't. Two relay processes fetching from the same pending rows without coordination will double-publish. You need either:

1. Advisory locks (Postgres pg_try_advisory_xact_lock) per row before processing—correct but adds per-row lock overhead
2. Claim-and-process: UPDATE outbox SET status='claimed', claimed_at=NOW() WHERE id = $1 AND status='pending' RETURNING *—optimistic, correct, but requires a claim timeout cleanup job for crashed workers
3. Partition the outbox by relay shard—relay A owns even aggregate_id hashes, relay B owns odd. Simple, no lock contention, but requires coordination on resharding

For most services, a single relay process with a liveness probe and fast restart is operationally simpler and safer than multi-relay coordination. Add throughput by batching publishes, not by parallelizing relay workers without coordination.

Decision Framework

Requirement	Polling Relay	WAL/CDC Relay
Latency tolerance > 1s	✓	✓
Latency < 500ms required	✗	✓
Operational simplicity priority	✓	✗
Multi-DB or managed DB (no WAL access)	✓	✗
High write volume, polling load concern	✗	✓

When to use per-aggregate sequence numbers: always, if any consumer downstream ever needs to reconstruct ordered state per entity. The cost is one SELECT MAX(sequence) FOR UPDATE per transaction; the benefit is consumer-side correctness that survives relay restarts and reordering.

When to skip the outbox entirely: if your broker supports transactional messaging natively (Kafka transactions, SQS FIFO with deduplication IDs and a two-phase approach), evaluate whether the dual-write risk is lower than the operational cost of maintaining an outbox. For most MongoDB + SNS/SQS stacks, the outbox remains the right call. For Postgres + Kafka with low broker latency requirements, WAL tailing to Kafka directly via Debezium may eliminate the outbox table as a separate concern.

The outbox pattern is correct. Its correctness is narrow. Design the relay, the schema, the consumer idempotency, and the ordering semantics explicitly rather than relying on the pattern's name to imply properties it does not provide.