system-design · beginner

Fault Tolerance — Keep Working When Parts Fail

The Central Question

Consider an enterprise financial processing engine running on the ReliabilityLab platform (reliabilitylab.com) processing 10,000,000 global transactions per day:

• Every non-trivial software application relies on multiple physical and logical components: application servers, database engines, network switches, caches, queues, and third-party Web APIs. Every single one of these components will eventually fail. Disks corrupt data, network hardware drops packets, memory leaks crash processes, and third-party APIs experience outages.
• In an unprepared system, the failure of a single background component stops the entire application. Customers see blank error pages, transactions fail midway through execution, and engineers are woken at 3:00 AM to perform manual restarts.

This lesson answers one central question: How do engineering teams build fault-tolerant architectures using N+1 redundancy, eliminate Single Points of Failure (SPOFs), and mathematically maximize system availability using MTBF and MTTR metrics?

Anatomy of a Failure: Faults, Errors, and Failures

To design fault-tolerant systems, engineers distinguish between three distinct stages of a breakdown: fault, error, and failure.

flowchart LR
  subgraph Component Level
    F[Fault: Root Cause] -->|triggers| E[Error: Invalid State]
  end
  subgraph System Boundary
    E -->|escapes boundary| FAIL[Failure: Broken Promise]
  end
  
  style F fill:#f9f9f9,stroke:#666,stroke-width:1px
  style E fill:#fff3cd,stroke:#ffebaa,stroke-width:1px
  style FAIL fill:#f8d7da,stroke:#f5c6cb,stroke-width:1px

Figure 1: The progression from an internal component fault to a user-visible service failure.

1. Fault

• Concrete example: A hard drive in a database server develops bad sectors, or a network fiber cable is accidentally cut by road construction.

2. Error

• Concrete example: The database process attempts to read a record, encounters an I/O read failure from the bad sector, and throws an internal StorageIOException in its log file.

3. Failure

• Concrete example: The web application cannot read customer account records from the database, returning an HTTP 500 Internal Server Error page to the customer during checkout.

The Objective of Fault Tolerance

Mathematical Formulation: Availability, MTBF, and MTTR

System availability ($A$) is mathematically defined as the ratio of operational uptime to total time, governed by Mean Time Between Failures (MTBF) and Mean Time To Repair (MTTR):

$$A = \frac{\text{MTBF}}{\text{MTBF} + \text{MTTR}}$$

gantt
    title System Availability Metrics Timeline
    dateFormat ss
    axisFormat %S
    section Operational Uptime
    MTBF: System Operational (1,000 Hours) :done, m1, 00, 50
    section Outage & Repair
    MTTR: Outage & Healing (1 Hour) :crit, r1, 50, 55
    section Operational Uptime
    MTBF: System Operational Restored :done, m2, 55, 105

Figure 2: System lifecycle showing operational MTBF uptime interspersed with MTTR repairs.

The Nines of Availability

Single Point of Failure (SPOF) Elimination

A Single Point of Failure (SPOF) is any isolated component whose failure causes the entire system to stop functioning.

flowchart LR
  subgraph Single Point of Failure Architecture
    Client1[User Browser] --> App1[Single Application Server]
    App1 --> DB1[(Single Primary Database)]
  end
subgraph Fault-Tolerant Redundant Architecture
    Client2[User Browser] --> LB[Load Balancer]
    LB --> AppA[App Instance A]
    LB --> AppB[App Instance B]
    AppA --> DBP[Primary DB]
    AppB --> DBP
    DBP -.->|Async Replication| DBS[Replica DB]
  end

Figure 3: Eliminating Single Points of Failure using load balancers and redundant replica servers.

Redundancy Models: Active-Passive vs. Active-Active

To survive node failures, architectures deploy physical or logical redundancy:

1. Active-Passive Redundancy (N+1 / 2N)

2. Active-Active Redundancy

Bulkheading & Fault Isolation Boundaries

Graceful Degradation & Feature Toggles

flowchart TD
  subgraph Active-Active Capacity Scaling
    LB[Load Balancer] --> Node1[Node 1: 33% Load]
    LB --> Node2[Node 2: 33% Load]
    LB --> Node3[Node 3: 33% Load]
    
    Node2 -.-x|Node 2 Fails!| Dead[CRASH]
    LB -->|Re-route Load| Node1
    LB -->|Re-route Load| Node3
    Note["Nodes 1 & 3 absorb load at 50% capacity each!"]
  end

Figure 4: Active-Active load redistribution upon node crash.

Complete Worked Example: Go Self-Healing Fault-Tolerant Worker Pool

Let's inspect a complete Go implementation of a self-healing fault-tolerant worker pool for the ReliabilityLab platform (reliabilitylab.com).

package main
import (
	"context"
	"fmt"
	"sync"
	"time"
)
type WorkerTask struct {
	ID    string
	Data  string
	Retry int
}
type SelfHealingWorkerPool struct {
	tasks      chan WorkerTask
	numWorkers int
	wg         sync.WaitGroup
}
func NewSelfHealingWorkerPool(numWorkers int, queueSize int) *SelfHealingWorkerPool {
	return &SelfHealingWorkerPool{
		tasks:      make(chan WorkerTask, queueSize),
		numWorkers: numWorkers,
	}
}
func (p *SelfHealingWorkerPool) Start(ctx context.Context) {
	for i := 1; i <= p.numWorkers; i++ {
		p.wg.Add(1)
		go p.runSupervisedWorker(ctx, i)
	}
}
func (p *SelfHealingWorkerPool) runSupervisedWorker(ctx context.Context, id int) {
	defer p.wg.Done()
for {
		select {
		case <-ctx.Done():
			return
		case task, ok := <-p.tasks:
			if !ok {
				return
			}
			p.executeWithRecovery(id, task)
		}
	}
}
func (p *SelfHealingWorkerPool) executeWithRecovery(workerID int, t WorkerTask) {
	defer func() {
		if r := recover(); r != nil {
			fmt.Printf("[SELF-HEALING] Worker %d Recovered from Panic on Task %s! Err: %v\n", workerID, t.ID, r)
			if t.Retry < 3 {
				t.Retry++
				p.tasks <- t // Re-enqueue for retry
			}
		}
	}()
// Execute task processing logic...
	if t.ID == "task_corrupt" && t.Retry == 0 {
		panic("CRITICAL_MEMORY_CORRUPTION_PANIC")
	}
fmt.Printf("[WORKER %d] Successfully Processed Task %s\n", workerID, t.ID)
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Faults are inevitable; failures are preventable: Distinguish between root cause faults, internal errors, and user-visible failures.
2. Eliminate all Single Points of Failure (SPOFs): Ensure every layer (DNS, Edge, App, Database, Queue) has redundant secondary instances.
3. Availability math depends on MTBF and MTTR: Maximize Mean Time Between Failures and minimize Mean Time To Repair ($A = \frac{\text{MTBF}}{\text{MTBF}+\text{MTTR}}$).
4. Provision N+1 headroom capacity: Active-Active clusters must maintain sufficient extra capacity to absorb full traffic load when a node fails.
5. Implement self-healing supervisor loops: Wrap application workers in panic recovery blocks and automated health monitors to self-heal without human intervention.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• Current setup: 2 API servers behind a load balancer connected to 1 primary SQL database.

1. Identify all Single Points of Failure (SPOFs) in the current setup.
2. Formulate an N+1 fault-tolerant architecture including primary-replica database failover and automated health probes.

Interactive Self-Assessment

A Fault is an underlying defect, an Error is an internal invalid state, and a Failure occurs when the error escapes to violate user contracts.

Faults refer to SQL schema bugs; Errors refer to network cables; Failures refer to client browser crashes.

Faults apply to hardware only; Errors apply to DNS servers; Failures apply to SSL certificates.

Errors are user-facing outage pages; Failures are background log messages; Faults are CPU clock speed reductions.

66% CPU (e.g. 80% CPU each), the failure of 1 node forces the remaining 2 nodes to absorb 120% CPU load each, causing both remaining nodes to crash sequentially in a domino cascading outage.">If 1 node fails, the remaining 2 nodes must absorb the lost node's traffic without exceeding 100% CPU capacity and crashing.

Running above 66% CPU automatically revokes client HTTPS TLS encryption certificates.

CPU levels above 66% force relational databases to drop B-Tree primary key indexes.

CPU levels above 66% trigger automatic hardware shutdown of local power generators.

What to Learn Next

• Failover — Seamless Transition When Primary Nodes Die: Explore active-passive failover and split-brain fencing.
• Disaster Recovery & RPO/RTO: Master recovery point objectives and regional disaster planning.
• Multi-Region Failover: Learn global server load balancing and cross-continent replication.

Track: Reliability and Operations

Previous: Failover — Switching to a Healthy Spare

Next: Heartbeats — Liveness Signals in Distributed Systems

Series: Reliability & Failover

1. Availability — Nines, Error Budgets, and Redundancy
2. Reliability — Correct Results Under Stress
3. Fault Tolerance — Keep Working When Parts Fail (this guide)
4. Failover — Switching to a Healthy Spare
5. Multi-Region Failover — Surviving a Region Outage
6. Disaster Recovery — RPO, RTO, and Backups That Work

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab