system-design · beginner

Data Replication — Keeping Copies in Sync

The Central Question

Consider a single PostgreSQL database server hosting an enterprise analytics platform on the DataLab platform (datalab.com) processing 100,000,000 queries per day:

• If the physical database server suffers a hardware failure (such as a motherboard failure or disk drive corruption), the entire application immediately collapses:

To eliminate single points of failure and scale read throughput, cloud architectures deploy Data Replication.

Data Replication is the process of copying data updates continuously from a primary database node to one or more secondary (replica) nodes over a network.

This lesson answers one central question: How do Single-Leader, Multi-Leader, and Leaderless database replication topologies maintain data copies across multiple machines, and how do synchronous vs asynchronous log shipping trade off commit latency against Recovery Point Objective (RPO) data loss during failover?

Single-Leader Topology: Primary and Read Replicas

The most common database replication topology in modern production architectures is Single-Leader Replication (Primary-Replica):

flowchart TB
  ClientW[Application Writers] -->|All Writes & Schema Updates| Primary[(Primary / Leader Node)]
  
  Primary -->|1. Write-Ahead Log Stream| R1[(Read Replica 1)]
  Primary -->|2. Write-Ahead Log Stream| R2[(Read Replica 2)]
  Primary -->|3. Write-Ahead Log Stream| R3[(Read Replica 3)]
  
  ClientR[Application Readers] -->|Read-Only Queries| R1
  ClientR -->|Read-Only Queries| R2
  ClientR -->|Read-Only Queries| R3

Figure 1: Single-Leader replication topology routing all writes to the Primary and reads to Read Replicas.

Operational Rules of Single-Leader Replication

1. All Writes Go to the Primary: Any INSERT, UPDATE, DELETE, or DDL schema modification must hit the Primary node. The Primary writes the change to its local Write-Ahead Log (WAL).
2. Replicas Apply Change Streams: Secondary nodes receive the WAL log stream from the Primary and apply changes sequentially to maintain identical data copies.
3. Read Distribution: Applications offload read-only SELECT queries across replicas, multiplying read throughput linearly.

Synchronous vs. Asynchronous Replication

How the Primary waits for replica acknowledgments before returning a COMMIT to the application client dictates the system's reliability and latency bounds:

sequenceDiagram
    autonumber
    actor App as Client Application
    participant Primary as Primary Node
    participant SyncRep as Sync Replica
    participant AsyncRep as Async Replica
    
    App->>Primary: 1. UPDATE balance = 500 (COMMIT)
    Primary->>Primary: 2. Write WAL to local disk
    
    par Synchronous Replication Path
        Primary->>SyncRep: 3. Forward WAL Log Record
        SyncRep->>SyncRep: 4. Write WAL to disk
        SyncRep-->>Primary: 5. ACK Log Received
    and Asynchronous Replication Path
        Primary->>AsyncRep: 6. Forward WAL Log Record (No Wait!)
    end
    
    Primary-->>App: 7. Commit Acknowledged (HTTP 200)
    Note over AsyncRep: 8. Async Replica receives WAL milliseconds later.

Figure 2: Sequence diagram contrasting synchronous replication waiting against asynchronous log shipping.

Multi-Leader and Leaderless (Dynamo) Topologies

When applications expand across multiple geographically distributed datacenters, Single-Leader replication introduces long-distance write latency. Systems deploy alternative topologies:

flowchart TD
  Topologies[Replication Topologies] --> SingleLeader[1. Single-Leader]
  Topologies --> MultiLeader[2. Multi-Leader / Active-Active]
  Topologies --> Leaderless[3. Leaderless / Dynamo]
  
  SingleLeader --> SLDesc["1 Primary accepts writes globally.<br/>Simple consistency; long-distance write latency."]
  MultiLeader --> MLDesc["1 Primary per Datacenter.<br/>Low write latency worldwide; complex write conflict resolution."]
  Leaderless --> LLDesc["No Primary. Clients write to N nodes simultaneously.<br/>Quorum Math (W + R > N) enforces consistency."]

Figure 3: Comparison of Single-Leader, Multi-Leader, and Leaderless topologies.

Semi-Synchronous Replication (Best of Both Worlds)

• The Primary requires acknowledgment from at least 1 synchronous replica before committing, while continuing asynchronous log shipping to 4 other read replicas.
• If the 1 synchronous replica fails, the Primary automatically degrades to asynchronous mode, preserving operational availability while bounding RPO data loss to $0$ as long as 1 replica remains healthy.

Read-After-Write & Monotonic Read Guarantees

• Read-After-Write Consistency: Guarantees that if a user modifies data (e.g. updating a bio), any immediate read query from that same user will return the updated data.
• Monotonic Reads: Guarantees that if a user reads a data state at time $T_1$, any subsequent reads will never return an older state from $T_0$ (preventing time travel anomalies).
• Consistent Prefix Reads: Guarantees that if writes occur in a specific causal order (Question $\rightarrow$ Answer), anyone reading the data will see the question before the answer.

Split-Brain Mitigation & Fencing Tokens

Physical WAL vs Logical Replication Streams

• Physical Replication: Streams raw byte-level disk block modifications from the Write-Ahead Log (WAL). Requires replicas to run identical major PostgreSQL database versions and OS architectures.
• Logical Replication: Streams decoded SQL transaction change events (INSERT INTO users ...). Allows cross-version database upgrades and selective table replication.

Leaderless Quorum Mathematics ($W + R > N$)

To guarantee that a read operation always sees the latest written value:

$$W + R > N$$

If $N = 5$, $W = 3$, and $R = 3$:

$$W + R = 3 + 3 = 6 > 5$$

At least 1 node in the read set is guaranteed to hold the latest timestamped write value.

Replication Lag Anomalies and Solutions

Asynchronous replication introduces a Replication Lag Window ($\Delta t_{\text{lag}}$). When users read from lagging replicas, three distinct consistency anomalies occur:

flowchart TD
  Anomalies[Replication Lag Consistency Anomalies] --> RYW[1. Reading Your Own Writes]
  Anomalies --> Mono[2. Monotonic Reads]
  Anomalies --> Prefix[3. Consistent Prefix Reads]
  
  RYW --> RYWDesc["User posts comment but cannot see it on refresh.<br/>Fix: Route user's reads to Primary for 5 seconds post-write."]
  Mono --> MonoDesc["User sees post, refreshes, and post disappears.<br/>Fix: Pin user sessions to a single replica."]
  Prefix --> PrefixDesc["User sees answer BEFORE the question.<br/>Fix: Enforce causal dependency order."]

Figure 4: Taxonomy of replication lag consistency anomalies and mitigations.

Complete Worked Example: Go Async Replication Lag Monitor

Let's inspect a complete Go implementation of an Async Replication Lag Monitor for the DataLab platform (datalab.com).

package main
import (
	"context"
	"fmt"
	"sync"
	"time"
)
type ReplicaHealth struct {
	ID           string
	PrimaryWAL   int64
	ReplicaWAL   int64
	LagBytes     int64
	LagDuration  time.Duration
	IsDegraded   bool
}
type ReplicationMonitor struct {
	mu       sync.RWMutex
	replicas map[string]*ReplicaHealth
	maxLag   time.Duration
}
func NewReplicationMonitor(maxLag time.Duration) ReplicationMonitor {
	return &ReplicationMonitor{
		replicas: make(map[string]ReplicaHealth),
		maxLag:   maxLag,
	}
}
func (m *ReplicationMonitor) UpdateReplicaStatus(id string, primaryWAL, replicaWAL int64, lagTime time.Duration) {
	m.mu.Lock()
	defer m.mu.Unlock()
lagBytes := primaryWAL - replicaWAL
	isDegraded := lagTime > m.maxLag
m.replicas[id] = &ReplicaHealth{
		ID:          id,
		PrimaryWAL:  primaryWAL,
		ReplicaWAL:  replicaWAL,
		LagBytes:    lagBytes,
		LagDuration: lagTime,
		IsDegraded:  isDegraded,
	}
if isDegraded {
		fmt.Printf("[REPLICATION ALERT] Replica %s DEGRADED! Lag: %v (Max Allowed: %v, Unapplied WAL: %d bytes)\n",
			id, lagTime, m.maxLag, lagBytes)
	}
}
func (m *ReplicationMonitor) GetHealthyReadReplicas() []string {
	m.mu.RLock()
	defer m.mu.RUnlock()
healthy := make([]string, 0)
	for id, rep := range m.replicas {
		if !rep.IsDegraded {
			healthy = append(healthy, id)
		}
	}
	return healthy
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Replication provides fault tolerance and read scale: Copy data across secondary nodes to survive hardware crashes and offload reads.
2. Synchronous guarantees zero RPO at latency cost: Synchronous replication waits for replica disk commits ($RPO = 0$), while Asynchronous offers sub-ms writes with lag risk.
3. Leaderless requires $W + R > N$ Quorum: Ensure read and write quorums overlap ($W + R > N$) to guarantee reading latest timestamped values.
4. Mitigate Replication Lag with Read-Your-Own-Writes: Route reads to the Primary for a few seconds following user updates to prevent stale reads.
5. Fence dead leaders with STONITH: Forcefully cut power or disk access from failed primary nodes before promoting read replicas.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• 50,000 read RPS, 2,000 write RPS.
• User requirement: Profile updates must be visible immediately to the editing user.

1. Formulate the replication topology (Single-Leader vs Multi-Leader) and log shipping method (Sync vs Async).
2. Design the read routing middleware to guarantee Read-Your-Own-Writes consistency for profile edits.

Interactive Self-Assessment

N) to guarantee reading the latest written data?">

N, the set of nodes written to (W) and the set of nodes read from (R) MUST overlap by at least 1 node. That overlapping node guarantees that the read set contains the newest timestamped write value.">When W + R > N, the write set (W) and read set (R) overlap by at least 1 node, ensuring reads include the latest write.

Quorum math converts relational database SQL schemas into flat text files.

Quorum math replaces public DNS nameservers with local hosts entries.

Quorum math doubles the physical hardware clock speed of primary database CPUs.

It guarantees zero data loss (RPO = 0), but increases write latency and halts writes if a synchronous replica fails.

Synchronous replication automatically formats persistent NVMe SSD disk drives on read replicas.

Synchronous replication revokes edge HTTPS TLS encryption certificates on load balancers.

Synchronous replication reboots operating system hypervisors across all replica nodes.

What to Learn Next

• Rebalancing Shards Under Skewed Traffic: Master live shard data migration and range splitting.
• Hot Partition / Hot Key — Mitigating Skewed Traffic: Learn key salting and local micro-caching.
• Database Sharding — Split Data Across Many Machines: Revisit shard keys and scatter-gather queries.

Track: Distributed Systems

Previous: Strong vs Eventual Consistency — Trade-offs in Distributed Systems

Series: Replication

1. Data Replication — Keeping Copies in Sync (this guide)
2. DDIA Notes — Replication & Partitioning
3. PostgreSQL Replication & Failover
4. Quorum Reads vs Quorum Writes

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab