system-design · beginner

Failover — Switching to a Healthy Spare

The Central Question

Consider a core enterprise database cluster running on the ReliabilityLab platform (reliabilitylab.com) processing 10,000,000 transactions per day:

• Primary Database Server (10.0.1.20): Receives 100% of incoming SQL write transactions.
• Standby Database Server (10.0.1.21): Continuously streams Write-Ahead Logs (WAL) from the primary in real time.

If the system lacks automated failover mechanics:

1. Application servers continue attempting to send SQL write transactions to 10.0.1.20.
2. Every transaction fails with a Connection Refused exception.
3. Even though the Standby Server (10.0.1.21) contains 100% of up-to-date data, it remains idle because no automated system promoted it to primary or updated application connection routes.
4. The platform suffers a 100% total outage lasting 3 hours until an engineer wakes up to execute manual shell scripts.

Failover is the automated operational state machine that detects component failures, promotes a healthy standby node, re-routes incoming traffic, and isolates the failed node to maintain continuous service availability.

This lesson answers one central question: How do automated failover systems execute the 4-stage control loop (Detect $\rightarrow$ Decide $\rightarrow$ Shift $\rightarrow$ Rejoin) using Quorum Consensus, Virtual IP (VIP) BGP Routing, and STONITH Fencing to achieve sub-minute RTO without triggering Split-Brain data corruption?

The 4-Stage Failover Control Loop

Every robust failover system operates as a continuous 4-stage control loop:

flowchart TD
  Detect[1. DETECT: Probe health & detect failure] --> Decide[2. DECIDE: Consensus election of new Leader]
  Decide --> Shift[3. SHIFT: Re-route Virtual IP & DNS traffic]
  Shift --> Rejoin[4. REJOIN: Fence & rebuild old Primary as Standby]
  Rejoin --> Detect

Figure 1: The 4-stage failover operational control loop.

Deconstructing the 4 Stages

1. Detect (Health Probes & Heartbeats):

1. Decide (Quorum Election & Consensus):

1. Shift (Traffic Re-Routing):

1. Rejoin (Fencing & Standby Provisioning):

Active-Passive vs. Active-Active Topologies

Failover architectures structure redundant nodes into two primary topologies:

flowchart TD
  FailoverTopologies[Failover Cluster Topologies] --> ActivePassive[1. Active-Passive / Active-Standby]
  FailoverTopologies --> ActiveActive[2. Active-Active Cluster]
  
  ActivePassive --> APDesc["ONE Primary node accepts writes.<br/>Standby sits idle or handles reads.<br/>Simple data consistency; non-zero RTO."]
  ActiveActive --> AADesc["ALL nodes accept writes simultaneously.<br/>Zero RTO failover.<br/>Requires complex multi-master conflict resolution."]

Figure 2: Architectural comparison of Active-Passive vs Active-Active topologies.

Split-Brain Syndrome and Fencing (STONITH)

The most catastrophic failure in a failover system occurs when a network partition isolates the primary node from the consensus monitors without shutting down the primary process:

flowchart TD
  subgraph Network Partition Split-Brain
    PrimaryNode[Old Primary Node 10.0.1.20] -.-x|Network Partition Cut| Monitor[Consensus Heartbeater]
    Monitor -->|Declares Primary Dead| StandbyNode[Standby Node 10.0.1.21]
    StandbyNode -->|Promoted to New Primary| NewPrimary[New Primary Node 10.0.1.21]
    
    ClientA[Client A] -->|Writes to Old Primary| PrimaryNode
    ClientB[Client B] -->|Writes to New Primary| NewPrimary
    
    style PrimaryNode fill:#f8d7da,stroke:#dc3545
    style NewPrimary fill:#f8d7da,stroke:#dc3545
  end

Figure 3: Split-Brain Syndrome causing dual primary write corruption.

Split-Brain Syndrome Mechanics

• Old Primary thinks it is still the legitimate leader and accepts writes from Client A.
• New Primary is promoted by consensus monitors and accepts writes from Client B.
• Result: Data diverges permanently across both nodes (Split-Brain Data Corruption). Restoring consistency requires manual database reconstruction.

The Solution: Fencing & STONITH

• STONITH ("Shoot The Other Node In The Head"): The failover controller uses intelligent Power Distribution Units (PDUs) or cloud IPMI APIs to physically cut power or revoke disk storage access from the old primary server before promoting the standby node.

Traffic Switching Mechanics: Virtual IP (VIP) Floating vs. DNS Failover

Shifting traffic to a new primary node requires network-level routing updates:

flowchart LR
  subgraph Virtual IP Floating ARP Shift
    Router[Network Switch / Router] -->|Virtual IP: 10.0.1.100| NodeA[Primary Node 10.0.1.20]
    NodeA -.-x|Fails| Dead[CRASH]
    Router -.->|Gratuitous ARP Update| NodeB[Standby Node 10.0.1.21]
    Note["VIP 10.0.1.100 shifts instantly in < 1 second!"]
  end

Figure 4: Instant Virtual IP floating using Gratuitous ARP updates.

1. Virtual IP (VIP) Floating (Sub-Second Failover)

• Both primary and standby nodes exist on the same local subnet.
• Clients connect to a shared Virtual IP (10.0.1.100).
• Upon failover, the new primary issues a Gratuitous ARP packet, instructing local network switches to route traffic for 10.0.1.100 to the new node's MAC address in $< 1\text{ second}$.

2. DNS-Based Failover (Multi-Region / Cross-Subnet)

• Updates DNS A records to point domain db.reliabilitylab.com to the new IP address.
• Limitation: DNS TTL caching delays failover for 30 to 300 seconds as client OS caches expire.

Consensus Election Mechanics (Raft / Etcd)

Leader Lease Timeouts & Clock Drift

Complete Worked Example: Go Automatic Failover Controller with Quorum Election

Let's inspect a complete Go implementation of an automatic failover controller for the ReliabilityLab platform (reliabilitylab.com).

package main
import (
	"context"
	"fmt"
	"sync"
	"time"
)
type NodeRole int
const (
	RoleStandby NodeRole = iota
	RolePrimary
)
type ClusterNode struct {
	ID       string
	Address  string
	Role     NodeRole
	IsAlive  bool
	WALOffset int64
}
type FailoverController struct {
	mu           sync.Mutex
	nodes        map[string]*ClusterNode
	activeLeader string
	quorumSize   int
}
func NewFailoverController(nodes map[string]ClusterNode) FailoverController {
	n := len(nodes)
	return &FailoverController{
		nodes:        nodes,
		activeLeader: "node-1",
		quorumSize:   (n / 2) + 1, // Quorum: (N/2)+1
	}
}
func (fc *FailoverController) MonitorLoop(ctx context.Context, checkInterval time.Duration) {
	ticker := time.NewTicker(checkInterval)
	defer ticker.Stop()
for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
			fc.evaluateClusterHealth()
		}
	}
}
func (fc *FailoverController) evaluateClusterHealth() {
	fc.mu.Lock()
	defer fc.mu.Unlock()
leader, exists := fc.nodes[fc.activeLeader]
	if exists && !leader.IsAlive {
		fmt.Printf("[ALERT] Primary Node %s is DEAD! Initiating Automated Failover...\n", fc.activeLeader)
		fc.executeFailover()
	}
}
func (fc *FailoverController) executeFailover() {
	// 1. STONITH Fencing: Fence old primary
	fmt.Printf("[STONITH FENCING] Hard-powering off dead primary %s...\n", fc.activeLeader)
	fc.nodes[fc.activeLeader].Role = RoleStandby
// 2. Quorum Election: Elect standby node with highest WALOffset
	var bestCandidate *ClusterNode
	aliveCount := 0
for _, node := range fc.nodes {
		if node.IsAlive {
			aliveCount++
			if bestCandidate == nil || node.WALOffset > bestCandidate.WALOffset {
				bestCandidate = node
			}
		}
	}
// Verify Quorum Consensus
	if aliveCount < fc.quorumSize {
		fmt.Printf("[FATAL FAILOVER ERROR] Cannot reach Quorum! Alive Nodes: %d / Required: %d. ABORTING FAILOVER.\n", aliveCount, fc.quorumSize)
		return
	}
// Promote New Leader
	bestCandidate.Role = RolePrimary
	fc.activeLeader = bestCandidate.ID
// 3. Shift Traffic: Virtual IP ARP update
	fmt.Printf("[SUCCESS] Promoted Node %s to PRIMARY! VIP Shifted to %s\n", bestCandidate.ID, bestCandidate.Address)
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Automated failover requires a 4-stage control loop: Detect failures $\rightarrow$ Decide consensus $\rightarrow$ Shift traffic $\rightarrow$ Rejoin old node as standby.
2. Prevent Split-Brain with STONITH Fencing: Power off or isolate the old primary node before promoting a standby node to prevent dual-master data corruption.
3. Enforce Majority Quorum Consensus: Quorum requires $Q = \lfloor \frac{N}{2} \rfloor + 1$ votes to elect a new leader and prevent split-brain partitions.
4. Use Virtual IP (VIP) Floating for instant shift: VIP floating via Gratuitous ARP updates shifts network traffic in $< 1\text{ second}$ without waiting for DNS TTL expirations.
5. Ensure hardware parity for standby nodes: Provision standby nodes with identical CPU, RAM, and IOPS specs as primary nodes to prevent overload crashes upon failover.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• 1 Primary Node (10.0.1.10) + 2 Standby Replicas (10.0.1.11, 10.0.1.12).

1. Calculate the Quorum size required for this 3-node PostgreSQL cluster.
2. Design the STONITH fencing sequence to guarantee zero split-brain writes when 10.0.1.10 becomes unreachable over the management network.

Interactive Self-Assessment

It guarantees the old primary is dead and cannot accept client writes, preventing Split-Brain data corruption.

It speeds up SQL B-Tree index creation on the standby node.

It automatically updates public DNS servers across all worldwide ISPs.

It compresses Write-Ahead Log files by up to 90% using Gzip.

It issues a Gratuitous ARP packet to local switches, shifting IP routing to the new node's MAC address instantly in <1s without DNS TTL delays.

It forces client browsers to clear their HTTPS TLS certificate caches.

It automatically restarts the operating system hypervisor on standby nodes.

It converts SQL database tables into un-indexed CSV disk files.

What to Learn Next

• Disaster Recovery & RPO/RTO: Master recovery point objectives, data backup strategies, and RTO reduction.
• Multi-Region Failover: Learn cross-continent Anycast routing and global server load balancing.
• Fault Tolerance — Keep Working When Parts Fail: Revisit single points of failure and availability percentage math.

Track: Reliability and Operations

Previous: Duplicate Requests & the Idempotency Gap

Next: Fault Tolerance — Keep Working When Parts Fail

Series: Reliability & Failover

1. Availability — Nines, Error Budgets, and Redundancy
2. Reliability — Correct Results Under Stress
3. Fault Tolerance — Keep Working When Parts Fail
4. Failover — Switching to a Healthy Spare (this guide)
5. Multi-Region Failover — Surviving a Region Outage
6. Disaster Recovery — RPO, RTO, and Backups That Work

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab