system-design · beginner

Strong vs Eventual Consistency — Trade-offs in Distributed Systems

The Central Question

Consider an international financial ledger and user management platform running on the TxLab platform (txlab.com) processing 100,000,000 queries per day across 50,000,000 active users in North America and Europe:

• A user updates an account balance from $100 to $80. The write is committed to the primary database in Virginia (us-east-1).

1. User A in New York queries the account balance.
2. User B in London queries the account balance.

If the architecture enforces Eventual Consistency, User A sees $80, but User B temporarily sees the stale balance of $100 for several hundred milliseconds until asynchronous replication catches up across regions.

This fundamental trade-off governs every distributed system: Should the architecture force readers to wait for global node synchronization (Strong Consistency), or allow temporary replica disagreement to maximize throughput and minimize latency (Eventual Consistency)?

This lesson answers one central question: How do distributed storage systems balance strict Linearizable Consistency against low-latency Eventual Consistency, and how do engineers design consistency models (including Read-Your-Writes, Monotonic Reads, and CRDT conflict resolution) across multi-region database architectures?

The Consistency Model Spectrum

Consistency is not a binary choice between "perfect" and "broken". Distributed databases operate along a continuous spectrum of guarantees:

flowchart LR
  Spectrum[Consistency Guarantee Spectrum]
  
  Spectrum --> Strongest[Linearizability / Strong Consistency]
  Spectrum --> Sequential[Sequential / Serializability]
  Spectrum --> Causal[Causal Consistency]
  Spectrum --> ClientCentric[Read-Your-Writes / Monotonic Reads]
  Spectrum --> Weakest[Eventual Consistency]
  
  style Strongest fill:#d4edda,stroke:#28a745
  style Weakest fill:#fff3cd,stroke:#ffc107

Figure 1: Spectrum of consistency models from strict Linearizability down to Eventual Consistency.

1. Strong Consistency (Linearizability)

2. Eventual Consistency

3. Intermediate Consistency Guarantees

Technical Mechanics: Quorums vs. Async Replication

The choice between strong and eventual consistency dictates the underlying network replication protocol:

sequenceDiagram
    autonumber
    actor Client as Client App
    participant Primary as Primary Node (US)
    participant Replica1 as Sync Replica (US)
    participant Replica2 as Async Replica (EU)
    
    Note over Client,Replica2: STRONG CONSISTENCY (Quorum Write: W + R > N)
    Client->>Primary: 1. WRITE Price = $80
    Primary->>Replica1: 2. Synchronous Replication
    Replica1-->>Primary: 3. ACK Write
    Primary-->>Client: 4. HTTP 200 OK (Strong Write Complete)
    
    Note over Client,Replica2: EVENTUAL CONSISTENCY (Async background shipping)
    Primary->>Replica2: 5. Asynchronous Log Shipping (Cross-Ocean WAN)
    Note over Replica2: 6. Replica EU receives update 250ms later.

Figure 2: Sequence diagram contrasting synchronous quorum writes against asynchronous background log shipping.

Read Repair and Anti-Entropy Background Synchronization

Eventual consistency models employ background repair processes to resolve replica divergence over time:

flowchart TD
  Convergence[Eventual Consistency Convergence Mechanisms] --> ReadRepair[1. Read Repair]
  Convergence --> AntiEntropy[2. Anti-Entropy with Merkle Trees]
  Convergence --> HintedHandoff[3. Hinted Handoff]
  
  ReadRepair --> RRDesc["When a client reads from multiple replicas (R > 1),<br/>if a stale node is detected, the client repairs it asynchronously."]
  AntiEntropy --> AEDesc["Background worker nodes exchange Merkle Tree hashes<br/>to quickly identify out-of-sync key ranges without scanning all rows."]
  HintedHandoff --> HHDesc["If Node C is temporarily offline during write,<br/>Node A stores a 'hint' and delivers it when Node C recovers."]

Figure 3: Taxonomy of background repair mechanisms enforcing eventual consistency.

Hinted Handoff Mechanics

Implementing Causal Consistency with Vector Clocks

• Each message payload includes a causal vector token (causal_dep: [M1]).
• When a client node receives $M_2$, it buffers $M_2$ in memory until $M_1$ arrives and is applied to the local database, guaranteeing that questions are always displayed before answers.

Tunable Consistency Levels (Cassandra / DynamoDB)

• Write LOCAL_QUORUM + Read LOCAL_QUORUM: Guarantees strong linearizable consistency within a single datacenter ($W + R > N$) with sub-10ms local network latency.
• Write ONE + Read ONE: Delivers maximum write and read throughput with sub-millisecond response times, accepting eventual consistency staleness.
• Write ALL: Guarantees absolute global consistency across all datacenters, but fails if any single node in the cluster goes offline.

Monotonic Clocks vs Wall Clocks in Consistency Engines

Client-Side Conflict Resolution Handlers

Complete Worked Example: Go Linearizable vs Eventual Consistency Simulator

Let's inspect a complete Go implementation of a Linearizable vs Eventual Consistency Simulator for the TxLab platform (txlab.com).

package main
import (
	"context"
	"fmt"
	"sync"
	"time"
)
type StorageNode struct {
	mu     sync.RWMutex
	value  string
	version int64
}
type ConsistencyCluster struct {
	nodes []*StorageNode
}
func NewConsistencyCluster(numNodes int) ConsistencyCluster {
	nodes := make([]StorageNode, numNodes)
	for i := 0; i < numNodes; i++ {
		nodes[i] = &StorageNode{value: "initial", version: 0}
	}
	return &ConsistencyCluster{nodes: nodes}
}
func (c *ConsistencyCluster) StrongWriteLinearizable(val string) {
	fmt.Printf("[STRONG WRITE] Synchronously updating all %d nodes...\n", len(c.nodes))
	var wg sync.WaitGroup
for i, node := range c.nodes {
		wg.Add(1)
		go func(n *StorageNode, idx int) {
			defer wg.Done()
			n.mu.Lock()
			n.value = val
			n.version++
			n.mu.Unlock()
		}(node, i)
	}
wg.Wait()
	fmt.Println("[STRONG WRITE] All nodes synchronized successfully (Linearizable).")
}
func (c *ConsistencyCluster) EventualWriteAsync(val string) {
	fmt.Println("[EVENTUAL WRITE] Updating Node 0 immediately; async shipping to others...")
	c.nodes[0].mu.Lock()
	c.nodes[0].value = val
	c.nodes[0].version++
	c.nodes[0].mu.Unlock()
// Async log shipping to secondary nodes with simulated WAN latency
	for i := 1; i < len(c.nodes); i++ {
		go func(n StorageNode, idx int) {
			time.Sleep(100  time.Millisecond) // Simulated WAN lag
			n.mu.Lock()
			n.value = val
			n.version++
			n.mu.Unlock()
			fmt.Printf("[ASYNC REPLICA %d] Eventually updated to '%s'\n", idx, val)
		}(c.nodes[i], i)
	}
}
func (c *ConsistencyCluster) ReadFromNode(nodeIdx int) string {
	c.nodes[nodeIdx].mu.RLock()
	defer c.nodes[nodeIdx].mu.RUnlock()
	return c.nodes[nodeIdx].value
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Consistency is a spectrum: Choose from Linearizability (Strong), Sequential, Causal, Read-Your-Writes, down to Eventual Consistency based on business requirements.
2. Strong Consistency costs latency and availability: Linearizability requires synchronous cross-node network roundtrips ($W + R > N$), increasing write latency.
3. Eventual Consistency maximizes throughput: Asynchronous replication delivers sub-millisecond local writes, but introduces temporary replica divergence windows.
4. Use Read-Your-Writes for user sessions: Route reads to the Primary or recent replica for a few seconds post-write to prevent user-perceived stale reads.
5. Repair replica divergence with Merkle Trees: Deploy background Anti-Entropy workers to exchange Merkle hashes and repair out-of-sync keys without scanning full datasets.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• Product Catalog Reviews (10,000 QPS): High read volume, tolerant of minor staleness.
• Inventory Stock Count (500 QPS): Zero tolerance for selling out-of-stock items.

1. Formulate the exact consistency model selection (Linearizability vs Eventual Consistency) for Product Reviews vs Inventory Stock.
2. Design the read-repair and quorum configuration for the Inventory Stock service.

Interactive Self-Assessment

N) before returning success to the client. This forces every write to wait for physical cross-ocean WAN network roundtrips, adding hundreds of milliseconds of latency.">Linearizability requires synchronous cross-region network roundtrips (W + R > N) to confirm write consensus before acknowledging the client.

Linearizability automatically formats persistent NVMe SSD disk drives on read replicas.

Linearizability revokes client HTTPS TLS encryption certificates on edge load balancers.

Linearizability cuts physical CPU hardware clock speeds in half across all database nodes.

It pins the editing user's reads to the Primary post-write, ensuring they see their own newly submitted content immediately without waiting for global replication.

Read-Your-Writes converts relational database primary key indexes into un-indexed CSV files.

Read-Your-Writes replaces public DNS nameservers with local hosts file entries.

Read-Your-Writes reboots operating system hypervisors across all application pods.

What to Learn Next

• Consensus Algorithms — Raft and Replicated State Machines: Learn Raft quorum leader elections.
• Active-Active Conflict Resolution — CRDTs and Vector Clocks: Master multi-region write conflict resolution.
• CAP Theorem & PACELC — Consistency vs Availability: Revisit network partition trade-offs.

Track: Distributed Systems

Previous: Consistency Models — From Strong to Eventual

Next: Data Replication — Keeping Copies in Sync

Series: CAP, Consistency & Quorums

1. CAP Theorem — Consistency, Availability, and Partition Tolerance
2. Consistency Models — From Strong to Eventual
3. Strong vs Eventual Consistency — Trade-offs in Distributed Systems (this guide)
4. CAP, Consistency & Idempotency
5. Quorum Reads vs Quorum Writes

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab