distributed-systems · intermediate

Transactional Outbox and Saga Patterns — Reliable Multi-Step Work

Start here

Two different problems show up as soon as you leave a single database transaction:

1. Dual-write problem: you update the database and publish a message. One can succeed while the other fails → lost updates or phantom events.
2. Multi-service workflow: placing an order touches inventory, payment, and shipping—each with its own database. You cannot wrap them in one simple ACID transaction across the network.

• Transactional outbox — write the event to an outbox table in the same DB transaction as business data; a publisher relays outbox rows to the broker later.
• Saga — a sequence of local transactions with compensating actions if a later step fails (cancel reservation, refund, etc.).

What you will learn

1. Explain the dual-write failure mode with a concrete timeline.
2. Implement the outbox idea step by step.
3. Define choreography vs orchestration sagas at a beginner level.
4. Work a complete order workflow example.
5. Know limitations (compensation is hard, not magic rollback).
6. Avoid claiming “exactly-once” without evidence.

What you should know first

Words you need before we begin

Simple story: registered mail ledger

You write in the office ledger “package 42 must go out” in the same notebook entry as “order 42 paid.” A clerk later walks the “to send” list to the post office. You never say “I paid” without a ledger line for mailing—and you never mail without a ledger.

Where the analogy stops: compensations (refunds) are messier than crossing out a line; partial failures need explicit design.

The problem: dual write without outbox

Timeline:

1. Begin DB transaction; insert order ord_9.
2. Commit order.
3. Publish OrderCreated to Kafka → network fails.
4. User sees success; search indexer never hears; warehouse silent.

1. Publish first.
2. DB commit fails.
3. Consumers process an order that does not exist.

Step-by-step: transactional outbox

Step 1 — Add an outbox table

Columns idea: id, aggregate_type, aggregate_id, event_type, payload, created_at, published_at.

Step 2 — Same transaction

BEGIN
  INSERT order ...
  INSERT outbox(event_type='OrderCreated', payload=...)
COMMIT

Step 3 — Relay publishes

Background worker:

• Select unpublished rows
• Publish to broker
• Mark published_at (or delete)

Step 4 — Consumers remain idempotent

Outbox does not remove at-least-once consumer needs.

Step 5 — Variants

• Polling publisher
• CDC (Change Data Capture) on outbox/order tables
• LISTEN/NOTIFY tricks (database-specific)

Step-by-step: saga idea

Step 1 — Break the business flow into local transactions

Example order:

1. Create order (pending)
2. Reserve inventory
3. Charge payment
4. Mark order confirmed / schedule shipment

Step 2 — Define success path and compensations

Step 3 — Choose coordination style

• Orchestrator service issues commands, tracks state machine.
• Choreography uses events (InventoryReserved → payment service listens).

Step 4 — Make every step idempotent and timeout-aware

Network retries will duplicate commands.

Step 5 — Accept compensation limits

You cannot un-send a real-world email perfectly; you send a correction. Money refunds take time. Sagas model business undo, not physics undo.

Visual mental model

sequenceDiagram
  participant API as Order API
  participant DB as Order DB
  participant OB as Outbox
  participant R as Relay
  participant K as Broker
  participant Inv as Inventory service
API->>DB: BEGIN order + outbox
  DB-->>API: COMMIT
  R->>OB: read unpublished
  R->>K: publish OrderCreated
  K->>Inv: deliver
  Inv->>Inv: reserve stock local TX

Learning question: Where is atomicity guaranteed—across order+outbox or across order+inventory?

Caption: Outbox atomizes notification intent with local data; sagas coordinate cross-service steps without global ACID.

Complete worked example: place order

Starting situation

Services: Order, Inventory, Payment. Each has its own database. User clicks Buy.

Constraints

• Never charge without a durable order intent
• Prefer not to oversell
• Payment provider supports idempotency keys
• Refunds allowed within minutes

Decisions

Execution (happy path)

1. Order DB: insert order + outbox OrderPlaced.
2. Relay publishes.
3. Orchestrator (or choreography) reserves inventory.
4. Charge with idempotency key ord_9.
5. Mark confirmed; publish OrderConfirmed.

Failure sample

Payment declines: release inventory, mark order PAYMENT_FAILED, notify user. No silent half-state without status.

Outcome

Clear states and compensations. Limitations: refund lag, manual ops for rare dual failures, more moving parts than a modular monolith transaction.

How it works in production

• Outbox libraries / Debezium CDC
• Saga state tables and timeouts
• Dashboards: outbox lag, unpublished age, saga stuck states
• Runbooks for “payment charged, order not confirmed”

Ownership

Order domain owns outbox for its events. Saga owner (orchestrator team) owns stuck-workflow alerts.

Failure modes

Trade-offs

Compare with related concepts

Common misunderstandings

1. “Outbox gives exactly-once processing.”

1. “Compensation is automatic rollback.”

1. “Sagas need microservices.”

1. “If publish is in the request thread after commit, we are fine.”

1. “One big distributed transaction is easier.”

Check your understanding

1. What is a dual write?
2. Why insert outbox rows in the same transaction as the order?
3. What is a compensating action?
4. Orchestration vs choreography in one sentence each?
5. Does outbox remove the need for consumer idempotency?

Practice

1. Draw a timeline where dual write loses an event.
2. Design outbox columns for PaymentCaptured.
3. Write saga steps + compensations for hotel booking (room + payment + email).
4. List metrics for outbox lag and stuck sagas.
5. Explain to a PM why “refund” is not always instant compensation.

Revision summary

• Outbox fixes dual-write between DB and broker using one local transaction.
• A relay publishes outbox rows asynchronously.
• Sagas coordinate multi-service steps with compensations.
• Everything is at-least-once until you add idempotency.
• These patterns add reliability machinery—and operational duty.

Glossary

Abbreviations and terminology

• ACID — Atomicity, Consistency, Isolation, Durability
• CDC — Change Data Capture
• 2PC — Two-Phase Commit
• DB — Database

What to learn next

1. Idempotency, retries, backoff
2. Exactly-once vs practical deduplication
3. CDC vs dual writes
4. Distributed transactions / saga deep dive

Deeper production notes

Measuring outbox health

Alert on age of oldest unpublished outbox row, not only queue depth. A stuck relay with small depth still blocks business notifications. Track publish error rate and duplicate publish rate separately.

Saga timeouts and human intervention

Every saga step needs a timeout and a terminal state: succeeded, compensated, or needs_ops. Pure automatic compensation sometimes cannot finish (provider outage during refund). Design an operator console for rare stuck workflows instead of pretending 100% automation.

Testing strategy

Contract tests for events; integration tests that kill the process between commit and publish (outbox should still relay); saga tests that fail step 3 and assert compensations for steps 1–2. Untested compensations fail for the first time in production.

FAQ from first-time learners

Q: Can the outbox live in Redis?
A: If Redis is not in the same atomic commit as your business DB, you reintroduce dual write. Prefer the same transactional store as the business row.

Q: Is a saga slower?
A: It can add latency and complexity, but it matches real multi-system workflows.

Q: Who reviews this in production?
A: Treat money and inventory sagas as specialist-reviewed designs; this lesson is conceptual, not a compliance manual.

Track: Distributed Systems

Previous: Distributed Locks and Consensus — Coordinating Without a Single Boss Memory

Next: Exactly-Once Processing vs Practical Deduplication

Series: Kafka & Event Streaming

1. Message Queues — Hand Work Off Without Blocking the User
2. Kafka Architecture — Brokers, Partitions, ISR, and Consumers
3. Kafka Partition Ordering and Delivery Guarantees
4. Kafka Exactly-Once Semantics — What Is Actually Guaranteed
5. Transactional Outbox and Saga Patterns — Reliable Multi-Step Work (this guide)
6. Kafka Streams Introduction
7. Kafka with Spring Boot Code Walkthrough

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab