system-design · intermediate

Multi-Region Failover — Surviving a Region Outage

The Central Question

Consider an enterprise financial processing platform running on the ReliabilityLab platform (reliabilitylab.com) processing 10,000,000 global transactions per day:

• The infrastructure is deployed across three cloud regions: us-east-1 (N. Virginia), eu-west-1 (Ireland), and ap-northeast-1 (Tokyo).
• At 4:00 PM EST, a major fiber backhaul cut and power grid collapse completely disable AWS us-east-1.

1. All North American users attempting to access api.reliabilitylab.com hit connection timeouts (HTTP 504 / Connection Refused).
2. Even though European (eu-west-1) and Asian (ap-northeast-1) cloud datacenters are 100% operational, North American traffic cannot reach them because global routing DNS points exclusively to us-east-1.
3. The company suffers a total regional blackout impacting 50% of global customers for 8 hours.

Multi-Region Failover is the architectural and networking capability to detect the degradation or destruction of an entire geographical cloud region and automatically re-route global user traffic to healthy secondary cloud regions with minimal disruption.

This lesson answers one central question: How do engineering teams architect Multi-Region Failover using Global Server Load Balancing (GSLB), Anycast BGP Routing, and Cross-Region Database Replication while managing WAN network latency and cross-continent data consistency?

The Physics of Distance: Cross-Region Latency & CAP Theorem

Deploying multi-region architectures requires overcoming the speed of light in fiber optic cables:

flowchart LR
  US[us-east-1: N. Virginia] <-->|70ms RTT Transatlantic Cable| EU[eu-west-1: Ireland]
  US <-->|160ms RTT Transpacific Cable| AP[ap-northeast-1: Tokyo]
  EU <-->|220ms RTT Cross-Eurasia| AP

Figure 1: Round-Trip Time (RTT) network latencies across global cloud regions.

The Replication Dilemma: Synchronous vs. Asynchronous

1. Synchronous Cross-Region Replication:

1. Asynchronous Cross-Region Replication:

Multi-Region Architectural Models

Multi-region failover structures global compute and storage into two primary patterns:

flowchart TD
  MultiRegionModels[Multi-Region Architectural Topologies] --> ActivePassive[1. Active-Passive Multi-Region]
  MultiRegionModels --> ActiveActive[2. Active-Active Multi-Region]
  
  ActivePassive --> APDesc["1 Primary Region handles ALL writes.<br/>Secondary Region receives async replication.<br/>Failover requires GSLB DNS shift + DB promotion."]
  ActiveActive --> AADesc["ALL Regions accept writes simultaneously.<br/>Local low latency worldwide.<br/>Requires Conflict-Free Replicated Data Types (CRDTs) or Spanner."]

Figure 2: Active-Passive vs Active-Active multi-region deployment topologies.

Active-Passive Multi-Region Topology

• Primary Region (us-east-1): Accepts 100% of global read/write traffic.
• Secondary Region (eu-west-1): Runs warm application worker pools and standby database replicas receiving WAL stream updates.
• Failover Trigger: When us-east-1 fails, GSLB updates DNS to direct global traffic to eu-west-1, and the standby database is promoted to primary.

Active-Active Multi-Region Topology

• Both Regions (us-east-1 & eu-west-1): Run fully operational API gateways and databases processing local user traffic concurrently.
• Failover Trigger: If us-east-1 fails, GSLB immediately diverts North American traffic to eu-west-1. Because eu-west-1 is already processing live writes, failover latency is $< 10\text{ seconds}$ (Zero RTO).

Traffic Redirection: GSLB vs. Anycast BGP Routing

Re-routing global user traffic during a regional outage relies on two core Internet routing technologies:

flowchart TD
  subgraph GSLB Health Check Failover
    GSLB[Global Server Load Balancer / Route53] -->|1. Health Check Fails| US[us-east-1 (FAILED)]
    GSLB -->|2. Redirect Traffic| EU[eu-west-1 (HEALTHY)]
    Client[Global User] -->|DNS Lookup: api.reliabilitylab.com| GSLB
    GSLB --x|Stop Returning IP| US
    GSLB -->|Return IP: 52.14.0.1| EU
  end

Figure 3: Global Server Load Balancer (GSLB) detecting regional failure and shifting DNS resolution.

1. Global Server Load Balancing (GSLB) / Latency DNS

• DNS providers (such as AWS Route53 or Cloudflare) execute continuous health checks against regional edge endpoints.
• If us-east-1 fails health probes, the GSLB removes us-east-1 IP addresses from DNS responses and returns eu-west-1 IP addresses.

2. Anycast BGP Routing (Sub-Second Network Failover)

• With Anycast, multiple physical datacenters across the world advertise the identical IP address (198.51.100.1) to Internet Service Providers via BGP (Border Gateway Protocol).
• When a regional datacenter dies, BGP routers automatically re-route IP packets to the next closest physical datacenter along the shortest BGP path in $< 1\text{ second}$, completely bypassing DNS TTL caching delays!

Complete Worked Example: Go Multi-Region GSLB Health Router

Let's inspect a complete Go implementation of a Multi-Region GSLB Health Router for the ReliabilityLab platform (reliabilitylab.com).

package main
import (
	"context"
	"fmt"
	"net/http"
	"sync"
	"time"
)
type RegionHealth struct {
	RegionCode    string
	EndpointURL   string
	IsHealthy     bool
	LatencyMs     int64
	ConsecutiveFails int
}
type GSLBRouter struct {
	mu           sync.RWMutex
	regions      map[string]*RegionHealth
	primaryRegion string
}
func NewGSLBRouter(primary string, regions map[string]RegionHealth) GSLBRouter {
	return &GSLBRouter{
		primaryRegion: primary,
		regions:       regions,
	}
}
func (g *GSLBRouter) MonitorRegions(ctx context.Context, interval time.Duration) {
	ticker := time.NewTicker(interval)
	defer ticker.Stop()
for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
			g.healthCheckAll()
		}
	}
}
func (g *GSLBRouter) healthCheckAll() {
	g.mu.Lock()
	defer g.mu.Unlock()
client := http.Client{Timeout: 2 * time.Second}
for code, reg := range g.regions {
		start := time.Now()
		resp, err := client.Get(reg.EndpointURL + "/health")
		
		if err == nil && resp.StatusCode == http.StatusOK {
			reg.IsHealthy = true
			reg.LatencyMs = time.Since(start).Milliseconds()
			reg.ConsecutiveFails = 0
		} else {
			reg.ConsecutiveFails++
			if reg.ConsecutiveFails >= 3 {
				reg.IsHealthy = false
			}
		}
fmt.Printf("[GSLB PROBE] Region %s | Healthy: %v | Latency: %dms | Fails: %d\n",
			code, reg.IsHealthy, reg.LatencyMs, reg.ConsecutiveFails)
	}
}
func (g *GSLBRouter) GetOptimalRegion(userContinent string) string {
	g.mu.RLock()
	defer g.mu.RUnlock()
// Check if primary regional node is healthy
	primary, exists := g.regions[g.primaryRegion]
	if exists && primary.IsHealthy {
		return primary.RegionCode
	}
// Failover: Find closest healthy alternative region
	fmt.Printf("[GSLB FAILOVER] Primary Region %s UNHEALTHY! Routing user from %s to backup region...\n", g.primaryRegion, userContinent)
	for code, reg := range g.regions {
		if reg.IsHealthy {
			return code
		}
	}
return "EMERGENCY_STATIC_MAINTENANCE_PAGE"
}

Failure Modes and Engineering Mitigations

Global Data Sovereignty and GDPR Compliance

Cross-Region Consensus Protocols (Google Spanner)

Multi-Master Conflict Resolution (LWW vs CRDTs)

What You Should Remember

1. Multi-Region Failover survives whole-datacenter disasters: Re-route global traffic away from an entire degraded cloud region to secondary datacenters.
2. Speed of light enforces latency trade-offs: Synchronous cross-region replication guarantees $\text{RPO} = 0$ but adds $70-160\text{ms}$ write latency; Asynchronous replication maintains $2\text{ms}$ write latency with $1-5\text{s}$ replication lag.
3. Use Anycast BGP for sub-second network failover: Anycast routes IP traffic at the network layer in $< 1\text{ second}$, bypassing client ISP DNS TTL caching delays.
4. Maintain 100% hardware parity in DR regions: Secondary regions must possess equal capacity to absorb total global traffic without crashing.
5. Use Stateless JWTs for session management: Store user session state in distributed caches or JWTs to prevent regional failover from logging out millions of users.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• Regions: us-east-1 (Virginia), eu-central-1 (Frankfurt), ap-southeast-1 (Singapore).
• Requirements: Sub-second failover for video playback, 5-minute RPO for user watch history.

1. Formulate the GSLB and Anycast network routing architecture for global video playback.
2. Detail how your database replication pipeline manages cross-continent replication lag across North America, Europe, and Asia.

Interactive Self-Assessment

The primary DB must wait for transoceanic network round-trips (70-160ms RTT) to confirm secondary region writes before responding.

Synchronous replication converts relational database schemas into flat text files.

Synchronous replication forces client browsers to clear their DNS cache files.

Synchronous replication cuts the physical memory RAM speed of database servers in half.

Multiple datacenters advertise the identical IP address via BGP; when a region fails, BGP routers re-route IP packets instantly in <1s.

Anycast forces client web browsers to re-generate their private TLS encryption keys.

Anycast automatically reboots operating system hypervisors across all regions.

Anycast converts SQL database tables into un-indexed CSV files.

What to Learn Next

• Retry Storms — When Recovery Makes the Outage Worse: Learn how client retries overload recovering multi-region clusters.
• Disaster Recovery & RPO/RTO: Revisit recovery point objectives and DR Game Day executions.
• Failover — Switching to a Healthy Spare: Review local cluster VIP floating and split-brain fencing.

Track: Staff+ Technical Leadership

Previous: Domain-Driven Design (DDD) Essentials

Next: Active-Active Conflict Resolution — CRDTs and Vector Clocks

Series: Reliability & Failover

1. Availability — Nines, Error Budgets, and Redundancy
2. Reliability — Correct Results Under Stress
3. Fault Tolerance — Keep Working When Parts Fail
4. Failover — Switching to a Healthy Spare
5. Multi-Region Failover — Surviving a Region Outage (this guide)
6. Disaster Recovery — RPO, RTO, and Backups That Work

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab