system-design · intermediate

Event-Driven Architecture — React to Facts, Don’t Chain Calls

The Central Question

Consider an enterprise e-commerce platform running on the MessageLab platform (messagelab.com) processing 100,000,000 asynchronous events per day:

• In a monolithic or tightly-coupled synchronous microservice architecture, completing an order requires an API Gateway to execute a complex sequential execution chain:

flowchart LR
  API[Checkout API] -->|1. Sync Call| Inv[Inventory Service]
  Inv -->|2. Sync Call| Pay[Payment Service]
  Pay -->|3. Sync Call| Fulfillment[Fulfillment Service]
  Fulfillment -->|4. Sync Call| Email[Email Service]
  Email -->|5. Sync Call| Analytics[Analytics Service]
  
  style API fill:#f8d7da,stroke:#dc3545

This synchronous call chain creates three severe architectural hazards:

1. Coupling Explosion: Every service must know the network address, API contracts, and internal logic of downstream dependencies.
2. Cascading Latency: Total latency equals the sum of all individual service response times ($\sum L_i$).
3. Availability Failure: If the Analytics Service experiences a minor outage, the entire payment checkout flow collapses.

Event-Driven Architecture (EDA) is a software architecture paradigm where state changes emit immutable notifications of past facts called Events, and autonomous microservices React to those facts asynchronously without central caller coordination.

This lesson answers one central question: How do engineering teams design Event-Driven Architectures using Event Choreography versus Workflow Orchestration, enforce atomic publishing via the Transactional Outbox pattern, and resolve distributed cross-service transactions using Sagas?

Deconstructing EDA: Events vs. Commands vs. Queries

Designing a clean Event-Driven Architecture requires strictly distinguishing three types of system messages:

flowchart TD
  Messages[Distributed Message Types] --> Cmd[1. Commands]
  Messages --> Query[2. Queries]
  Messages --> Event[3. Events]
  
  Cmd --> CmdDesc["'ReserveInventory'<br/>Targeted Intent / Imperative Action.<br/>Directed to a SINGLE receiver.<br/>Can be REJECTED or fail."]
  Query --> QueryDesc["'GetOrderStatus'<br/>Request for Current State.<br/>Synchronous read path.<br/>Side-effect free."]
  Event --> EventDesc["'OrderPlaced'<br/>Immutable Past Fact.<br/>Broadcast to MULTIPLE subscribers.<br/>CANNOT be rejected!"]

Figure 1: Taxonomy of Commands, Queries, and Domain Events.

Key Property of Domain Events

Because an event describes the past (e.g. OrderPlaced, PaymentFailed, ShipmentDispatched), it cannot be rejected or cancelled by downstream subscribers. Subscribers can only react to the fact by updating their local state or executing secondary side-effects.

Event Composition: Choreography vs. Workflow Orchestration

When a business process spans multiple microservices, EDA organizes interactions using two primary coordination patterns:

flowchart TD
  subgraph Event Choreography: Decentralized Reactions
    C1[Order Service: OrderPlaced] -->|Broadcast Event| C2[Payment Service: PaymentCharged]
    C2 -->|Broadcast Event| C3[Inventory Service: ItemsReserved]
    C3 -->|Broadcast Event| C4[Fulfillment Service: OrderShipped]
  end
subgraph Workflow Orchestration: Centralized Controller
    O1[Saga Orchestrator] -->|1. Command: ChargePayment| O2[Payment Service]
    O1 -->|2. Command: ReserveInventory| O3[Inventory Service]
    O1 -->|3. Command: ShipOrder| O4[Fulfillment Service]
  end

Figure 2: Event Choreography (decentralized reactions) vs Workflow Orchestration (centralized orchestrator).

The Dual-Write Problem & The Transactional Outbox Pattern

A fundamental reliability trap in Event-Driven Systems occurs when an application attempts to write to a local database AND publish an event to a message broker in the same API request (The Dual-Write Problem):

flowchart TD
  App[Order Service API] -->|Step 1: SQL INSERT INTO orders| DB[(Local PostgreSQL DB)]
  App -->|Step 2: Publish Event to Kafka| Kafka((Kafka Message Broker))
  
  Note1["CRASH HAZARD! If step 1 succeeds but app crashes before step 2,<br/>the database has the order, but Kafka NEVER receives the event!"]

Figure 3: The Dual-Write Problem leading to catastrophic state inconsistency.

The Solution: The Transactional Outbox Pattern

flowchart TD
  App[Order Service API] -->|1. Single Local ACID Transaction| DB[(PostgreSQL Database)]
  
  subgraph Single Local ACID Transaction Boundary
    DB --> T1[INSERT INTO orders ...]
    DB --> T2[INSERT INTO outbox_events ...]
  end
  
  Poller[Outbox Relay Poller / Debezium CDC] -->|2. Poll Unpublished Outbox Rows| T2
  Poller -->|3. Publish Event to Broker| Kafka((Kafka Broker))
  Poller -->|4. Mark Outbox Row as Processed| T2

Figure 4: The Transactional Outbox Pattern embedding event records into local database ACID transactions.

Saga Orchestration & Compensating Transactions

• A Saga is a sequence of local transactions executed across independent microservices.
• Each local transaction updates a local database and emits a domain event triggering the next step.
• If a step fails (e.g. PaymentService returns InsufficientFunds), the Saga executes a series of Compensating Transactions in reverse order (e.g. UnreserveInventory, CancelOrder) to roll back business state safely (Eventual Consistency).

Event Sourcing Architecture

• Current State Derivation: Current entity state is calculated on demand by replaying the event log from the beginning of time (0 + 1000 - 500 = 500).
• Complete Audit Trail: Event sourcing provides 100% complete historical auditing, time-travel debugging, and the ability to rebuild read models at any point in history.

CQRS (Command Query Responsibility Segregation) Architecture

• Write Side (Command Path): Optimized for fast sequential writes and transaction validation using normalized SQL databases or event stores.
• Read Side (Query Path): Optimized for high-speed client queries using denormalized read models stored in Elasticsearch, Redis, or DynamoDB.
• Asynchronous Projection Sync: Event streams continuously project write-side state updates onto the read-side data stores in real time.

Outbox Relay Architecture: Polling Publisher vs CDC Publisher

• Polling Publisher: A background worker process periodically queries the database (SELECT * FROM outbox_events WHERE processed = false LIMIT 100 FOR UPDATE), publishes the events to Kafka, and marks the rows as processed. While simple, polling introduces database CPU query overhead under high concurrency.
• Transaction Log Miner (CDC Publisher): Debezium mines the database WAL log directly for new inserts into the outbox_events table. This delivers sub-10ms event publishing latency with zero database polling CPU overhead.

Event Replay and Time-Travel Debugging

• If a critical bug corrupts downstream analytics calculations over the weekend, developers deploy a patched version of the analytics service.
• The service resets its Kafka consumer group offset back to Friday night (--to-datetime "2026-07-20T00:00:00Z").
• It replays all historical events from the stream log, regenerating perfect read models without requiring database backups or manual data entry.

Event Envelope Standards (CloudEvents Specification)

• Envelopes mandate uniform top-level attributes: specversion, type, source, id, time, and datacontenttype.
• This ensures universal tracing, logging, and routing compatibility across all event-driven microservices regardless of programming language.

Microservice Event Contract Governance

Complete Worked Example: Production Go Transactional Outbox Event Producer

Let's inspect a complete Go implementation of a Transactional Outbox Event Producer for the MessageLab platform (messagelab.com).

package main
import (
	"context"
	"database/sql"
	"encoding/json"
	"fmt"
	"time"
)
type Order struct {
	ID        string  json:"order_id"
	UserID    string  json:"user_id"
	Amount    float64 json:"amount"
	Status    string  json:"status"
	CreatedAt time.Time json:"created_at"
}
type OutboxEvent struct {
	ID          string    json:"id"
	AggregateType string  json:"aggregate_type"
	AggregateID string    json:"aggregate_id"
	EventType   string    json:"event_type"
	Payload     string    json:"payload"
	Processed   bool      json:"processed"
	CreatedAt   time.Time json:"created_at"
}
type OrderService struct {
	db *sql.DB
}
func NewOrderService(db sql.DB) OrderService {
	return &OrderService{db: db}
}
func (s *OrderService) CreateOrderAtomic(ctx context.Context, order Order) error {
	// Start Single Local ACID Transaction
	tx, err := s.db.BeginTx(ctx, nil)
	if err != nil {
		return fmt.Errorf("begin transaction failed: %w", err)
	}
	defer tx.Rollback()
// 1. Insert Order Row into Business Table
	_, err = tx.ExecContext(ctx,
		"INSERT INTO orders (id, user_id, amount, status, created_at) VALUES ($1, $2, $3, $4, $5)",
		order.ID, order.UserID, order.Amount, order.Status, order.CreatedAt)
	if err != nil {
		return fmt.Errorf("insert order failed: %w", err)
	}
// 2. Serialize Order Payload for Event
	payloadBytes, err := json.Marshal(order)
	if err != nil {
		return fmt.Errorf("json marshal failed: %w", err)
	}
// 3. Insert Event Row into Outbox Table (SAME ACID TRANSACTION!)
	outboxID := fmt.Sprintf("evt_%s", order.ID)
	_, err = tx.ExecContext(ctx,
		"INSERT INTO outbox_events (id, aggregate_type, aggregate_id, event_type, payload, processed, created_at) VALUES ($1, $2, $3, $4, $5, $6, $7)",
		outboxID, "Order", order.ID, "OrderCreated", string(payloadBytes), false, time.Now())
	if err != nil {
		return fmt.Errorf("insert outbox event failed: %w", err)
	}
// 4. Commit Local Transaction
	if err := tx.Commit(); err != nil {
		return fmt.Errorf("transaction commit failed: %w", err)
	}
fmt.Printf("[TRANSACTIONAL OUTBOX SUCCESS] Order %s and Outbox Event %s committed atomically to local DB.\n",
		order.ID, outboxID)
	return nil
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. EDA structures software around immutable past facts: Emit events when state changes occur and let microservices react asynchronously.
2. Never execute Dual-Writes: Avoid updating a database and publishing to a message broker in separate network calls; use the Transactional Outbox Pattern.
3. Use Choreography for simple flows, Orchestration for complex sagas: Use Choreography for 2-3 service reactions; use central Saga Orchestrators for complex multi-step transactions.
4. Prefer Event-Carried State Transfer (ECST): Include self-contained data in event payloads to eliminate downstream HTTP back-queries and RPC coupling.
5. Enforce At-Least-Once Consumer Idempotency: Design subscriber handlers to handle duplicate event deliveries safely using unique event ID ledgers.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• OrderPlacedEvent triggers Inventory Reservation, Payment Authorization, and Warehouse Dispatch.
• If Payment Authorization fails, Inventory Reservation must be cancelled.

1. Formulate the Saga pattern choice (Choreography vs Orchestration) and compensating transaction flow for payment failure.
2. Detail how your architecture enforces the Transactional Outbox pattern to guarantee zero lost events.

Interactive Self-Assessment

It embeds the event payload into a local 'outbox_events' table inside the exact same local ACID transaction as business data.

It automatically formats persistent NVMe SSD disk drives on message broker servers.

It revokes edge HTTPS TLS encryption certificates on load balancers.

It doubles the physical hardware clock speed of primary database CPUs.

Choreography uses decentralized, autonomous microservice reactions; Orchestration uses a central coordinator issuing commands.

Choreography automatically converts relational database primary key indexes into un-indexed CSV files.

Orchestration replaces public DNS nameservers with local hosts file entries.

Choreography reboots operating system hypervisors across all microservice nodes.

What to Learn Next

• Change Data Capture (CDC): Learn log-based WAL database event streaming without outbox tables.
• Dead-Letter Queues & Poison Messages: Master poison message quarantine workflows.
• Publish-Subscribe (Pub-Sub) — Fan-Out Event Distribution: Revisit topic fan-out and consumer groups.

Track: Data, Storage and Messaging

Previous: Dead-Letter Queues and Poison Messages — Quarantine Work That Keeps Failing

Next: Publish/Subscribe Messaging — One Event, Many Interested Listeners

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab