system-design · intermediate

Active-Active Conflict Resolution — CRDTs and Vector Clocks

The Central Question

Consider a multi-region collaborative document and user profile platform running on the TxLab platform (txlab.com) processing 100,000,000 queries per day:

• The platform operates Active-Active primary databases in Region US-East (Virginia) and Region EU-West (Frankfurt).
• User 1 in New York updates their profile status to "Active - Working from Home" on the US primary database at 12:00:00.100.
• Simultaneously, User 1's mobile app in London automatically updates their profile status to "Active - In Transit" on the EU primary database at 12:00:00.105.

If the system naively applies the write that arrives second, network jitter could cause older updates to overwrite newer updates, or cause US and EU database nodes to diverge permanently.

This condition is an Active-Active Write Conflict.

This lesson answers one central question: How do Active-Active multi-master databases detect and resolve concurrent write conflicts across regions, and how do engineers compare Last-Write-Wins (LWW), Vector Clocks, and Conflict-Free Replicated Data Types (CRDTs) to achieve deterministic eventual convergence?

The Active-Active Multi-Master Topology

In Active-Active database architectures, every regional data center hosts a primary database node capable of accepting read and write transactions locally:

flowchart TD
  subgraph Region US-East Datacenter
    ClientUS[Client US] -->|1. Local Write: status = 'Working'| PrimaryUS[(Primary US DB)]
  end
subgraph Region EU-West Datacenter
    ClientEU[Client EU] -->|1. Local Write: status = 'Transit'| PrimaryEU[(Primary EU DB)]
  end
PrimaryUS <-->|2. Cross-Ocean Async Replication Stream| PrimaryEU
  
  Note over PrimaryUS,PrimaryEU: WRITE CONFLICT! Both nodes modified 'user:101' concurrently!

Figure 1: Active-Active multi-master topology allowing concurrent local writes across regions.

Conflict Resolution Strategies: LWW, Vector Clocks, and CRDTs

Databases deploy three principal mechanisms to resolve concurrent write conflicts:

flowchart TD
  Strategies[Active-Active Conflict Resolution] --> LWW[1. Last-Write-Wins LWW]
  Strategies --> VC[2. Vector Clocks / Version Vectors]
  Strategies --> CRDT[3. Conflict-Free Replicated Data Types CRDT]
  
  LWW --> LWWDesc["Uses physical NTP timestamp.<br/>Highest timestamp wins; risks silent data loss due to clock skew."]
  VC --> VCDesc["Tracks causal history per node [US:2, EU:1].<br/>Detects true concurrency and prompts application merge."]
  CRDT --> CRDTDesc["Mathematical data structures (PN-Counters, LWW-Element-Set).<br/>Guarantees deterministic convergence without locks."]

Figure 2: Taxonomy of active-active conflict resolution strategies.

Strategy 1: Last-Write-Wins (LWW) & The Clock Skew Risk

The simplest conflict resolution strategy is Last-Write-Wins (LWW):

• Each write payload includes a physical wall-clock timestamp (e.g. 1721820000100ms).
• When a database node receives a conflicting write, it compares timestamps and preserves the write holding the higher timestamp, discarding the older write.

The Fatal Vulnerability: Clock Skew Data Loss

If Server US's physical clock is 20ms ahead of Server EU's clock, a write executed on Server EU at 12:00:00.050 will be silently overwritten and discarded by an earlier write executed on Server US at 12:00:00.040, causing silent data loss!

Strategy 2: Vector Clocks (Causal History Tracking)

To detect true concurrency without relying on physical NTP clock synchronization, systems use Vector Clocks:

• A Vector Clock is an array of logical counters, with one counter per node: $V = [N_1: c_1, N_2: c_2, \dots, N_k: c_k]$.
• When Node A executes a write, it increments its own counter ($V[\text{NodeA}]++$) and attaches the vector clock to the data payload.

flowchart LR
  subgraph Causal Ordering vs Concurrency
    V1["Vector A: [US:1, EU:0]"] -->|Causally Precedes| V2["Vector B: [US:2, EU:0]"]
    
    V3["Vector C: [US:2, EU:0]"] <-->|CONCURRENT CONFLICT!| V4["Vector D: [US:1, EU:1]"]
  end

Figure 3: Comparing causally dominant vector clocks against concurrent conflicting vectors.

If Vector A has higher or equal counters than Vector B across all entries, Vector A causally succeeds Vector B (overwrite is safe). If neither vector dominates, a True Concurrent Conflict is flagged, prompting application code or user intervention to merge.

Strategy 3: Conflict-Free Replicated Data Types (CRDTs)

For complex data structures (such as shopping carts, online user counters, or collaborative text documents), systems deploy Conflict-Free Replicated Data Types (CRDTs).

CRDTs are mathematically designed data structures that can be updated independently across multiple regional nodes without central coordination, guaranteeing that as soon as all nodes receive the same set of updates, they automatically converge to the exact same state.

Mathematical Foundations of CRDTs

1. Commutative: $A \sqcup B = B \sqcup A$ (Order of update arrival does not matter).
2. Associative: $(A \sqcup B) \sqcup C = A \sqcup (B \sqcup C)$ (Grouping of network packets does not matter).
3. Idempotent: $A \sqcup A = A$ (Duplicate network packet delivery does not alter state).

State-Based (CvRDT) vs Operation-Based (CmRDT)

• State-Based CRDTs (CvRDT): Replicas continuously send their full state payload to peer nodes. Peer nodes apply the lattice merge function ($\sqcup$). CvRDTs are resilient to duplicate or lost network packets, but transmit larger payload sizes over WAN connections.
• Operation-Based CRDTs (CmRDT): Replicas transmit concurrent delta operations (e.g. add(item_42)) over an exact-once, causal reliable broadcast channel. CmRDTs transmit smaller network payloads, but require strict causal messaging delivery guarantees.

Vector Clock Garbage Collection & Pruning

• If a vector entry has remained unchanged for longer than $T_{\text{gc}}$ (e.g. 14 days), background compaction workers prune old node entries.
• If an old node re-appears after pruning, the system safely falls back to a deterministic LWW heuristic.

LWW-Element-Set Removal Bias vs Add Bias

• Add-Bias Set: Resolves equal timestamp ties in favor of keeping the item in the set.
• Remove-Bias Set: Resolves equal timestamp ties in favor of removing the item, preventing deleted items from accidentally reappearing.

CRDT State Compression

Complete Worked Example: Go CRDT Conflict Resolver (PN-Counter & LWW-Set)

Let's inspect a complete Go implementation of a Conflict-Free Replicated Data Type (P-N Counter and LWW-Element-Set) for the TxLab platform (txlab.com).

package main
import (
	"fmt"
	"sync"
)
// PNCounter is a State-Based Positive-Negative Counter CRDT
type PNCounter struct {
	mu        sync.RWMutex
	nodeID    string
	posCounts map[string]int64
	negCounts map[string]int64
}
func NewPNCounter(nodeID string) *PNCounter {
	return &PNCounter{
		nodeID:    nodeID,
		posCounts: make(map[string]int64),
		negCounts: make(map[string]int64),
	}
}
func (c *PNCounter) Increment(amount int64) {
	c.mu.Lock()
	defer c.mu.Unlock()
	c.posCounts[c.nodeID] += amount
}
func (c *PNCounter) Decrement(amount int64) {
	c.mu.Lock()
	defer c.mu.Unlock()
	c.negCounts[c.nodeID] += amount
}
func (c *PNCounter) Value() int64 {
	c.mu.RLock()
	defer c.mu.RUnlock()
var total int64
	for _, v := range c.posCounts {
		total += v
	}
	for _, v := range c.negCounts {
		total -= v
	}
	return total
}
func (c PNCounter) Merge(other PNCounter) {
	c.mu.Lock()
	defer c.mu.Unlock()
	other.mu.RLock()
	defer other.mu.RUnlock()
// Merge Positive Counts (Take Max per node)
	for k, v := range other.posCounts {
		if v > c.posCounts[k] {
			c.posCounts[k] = v
		}
	}
// Merge Negative Counts (Take Max per node)
	for k, v := range other.negCounts {
		if v > c.negCounts[k] {
			c.negCounts[k] = v
		}
	}
	fmt.Printf("[CRDT MERGE] Nodes merged cleanly. Consolidated Value: %d\n", c.Value())
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Active-Active enables multi-region local writes: Allow clients in US and EU to write locally for sub-10ms response times, but prepare for write conflicts.
2. LWW is simple but vulnerable to clock skew: Last-Write-Wins discards older timestamps, risking silent data loss when physical server clocks drift.
3. Vector Clocks track causal history: Use arrays of logical counters per node to detect true concurrent conflicts without relying on physical NTP clocks.
4. CRDTs guarantee deterministic mathematical convergence: Use CRDTs (PN-Counters, LWW-Sets, OR-Sets) to merge divergent replica states automatically without locks.
5. CRDT merge functions must be Commutative, Associative, and Idempotent: Ensure merge functions ($A \sqcup B$) handle out-of-order and duplicate network packet arrivals cleanly.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• Users in US, EU, and Asia edit the same shared document simultaneously.
• Critical requirement: Document edits must converge deterministically across regions without data loss.

1. Formulate the exact CRDT data structure selection (Sequence CRDT / Ropes vs LWW-Set) for real-time document editing.
2. Design the vector clock pruning mechanism to prevent metadata bloat as millions of clients join and leave document sessions.

Interactive Self-Assessment

Physical server clocks experience Clock Skew across regions, causing writes on lagging clocks to receive older timestamps and be silently discarded.

LWW causes physical hardware electrical short-circuits in server power units.

LWW revokes client HTTPS TLS encryption certificates on edge load balancers.

LWW doubles the physical hardware clock speed of primary database CPUs.

Commutative (order doesn't matter), Associative (grouping doesn't matter), and Idempotent (duplicates don't matter).

CRDT merge functions automatically convert relational database primary key indexes into un-indexed CSV files.

CRDT merge functions replace public DNS nameservers with local hosts file entries.

CRDT merge functions reboot operating system hypervisors across all cluster nodes.

What to Learn Next

• Consensus Algorithms — Raft and Replicated State Machines: Review single-leader Raft quorum elections.
• Strong vs Eventual Consistency: Revisit linearizability vs eventual consistency models.
• CAP Theorem & PACELC — Consistency vs Availability: Revisit network partition trade-offs.

Track: Staff+ Technical Leadership

Previous: Multi-Region Failover — Surviving a Region Outage

Next: Payment Platform Architecture

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab