system-design · intermediate

Circuit Breakers and Cascading Failure Control — Stop One Fire From Burning the Block

The Central Question

Consider a microservice platform running on the TrafficLab platform (trafficlab.com) processing 50,000 requests per second:

• When a customer clicks "Place Order", the Checkout Service executes synchronous HTTP calls to three internal microservices:

1. Inventory Service (reserves stock items)
2. Payment Service (processes credit card charges)
3. Recommendation Service (fetches personalized "You Might Also Like" upsell items)

Because the Checkout Service uses a shared thread pool of 100 worker threads to handle HTTP requests, incoming checkout calls begin waiting 30 seconds for recommendation responses. Within 5 seconds, all 100 worker threads become blocked, waiting on the slow recommendation service.

When a 101st customer attempts to place an order, the Checkout Service rejects the connection (HTTP 503 Thread Pool Exhausted). Even though the Payment Service and Inventory Service are 100% healthy, customers cannot complete purchases.

A localized failure in a non-essential recommendation widget has triggered a total platform collapse—a Cascading Failure.

To isolate network dependencies and contain failure blast radiuses, distributed architectures deploy Circuit Breakers.

A Circuit Breaker is a software state machine wrapped around network client calls that measures error rates over a sliding time window. When downstream failure rates exceed a threshold, the breaker "trips open", short-circuiting subsequent calls to fail fast locally without sending network packets.

This lesson answers one central question: How do circuit breakers transition between Closed, Open, and Half-Open states using sliding-window error metrics, and how do engineers combine breakers with Timeouts and Bulkheads to eliminate cascading outages?

Anatomy of a Cascading Failure

A Cascading Failure occurs when a localized fault in a single downstream dependency propagates upstream, causing successive healthy components to starve of resources and fail.

sequenceDiagram
    autonumber
    actor User as Client Browser
    participant API as Checkout Gateway (100 Threads)
    participant Pay as Payment Service (HEALTHY)
    participant Rec as Recommendation Service (UNHEALTHY)
    
    User->>API: 1. POST /checkout
    
    rect rgb(240, 255, 240)
        Note over API,Pay: Healthy Core Path
        API->>Pay: 2. POST /v1/charges
        Pay-->>API: 3. HTTP 200 OK (50ms)
    end
    
    rect rgb(255, 240, 240)
        Note over API,Rec: Un-bounded Non-Essential Path
        API->>Rec: 4. GET /v1/recommendations
        Note over Rec: DB Lockup! Hangs for 30,000ms!
        Note over API: Thread #1 BLOCKS waiting for 30s!
    end
    
    Note over API: Requests #2 to #100 BLOCK on Recommendation Service!
    
    User->>API: 5. POST /checkout (101st Request)
    API-->>User: 6. HTTP 503 Service Unavailable (Thread Pool Starvation!)
    Note over API: Core Payment Flow CRASHED by optional recommendation widget!

Figure 1: Sequence diagram demonstrating thread pool starvation causing a cascading failure.

The Mathematical Cascade Mechanism

If incoming traffic arrives at rate $\lambda$ requests/sec, and the average response latency of a downstream dependency increases from $L_{\text{healthy}}$ to $L_{\text{unhealthy}}$, Little's Law governs the required concurrent thread capacity $L_{\text{req}}$:

$$L_{\text{req}} = \lambda \times L_{\text{unhealthy}}$$

If $\lambda = 50 \text{ req/sec}$ and $L_{\text{unhealthy}} = 10 \text{ seconds}$:

$$L_{\text{req}} = 50 \times 10 = 500 \text{ threads}$$

If the thread pool size is $N_{\text{threads}} = 100$, the pool exhausts in exactly 2 seconds ($100 / 50$), causing all subsequent incoming API requests to fail with HTTP 503!

Circuit Breaker State Machine Mechanics

A circuit breaker operates as a 3-state finite state machine wrapped around a network call client:

stateDiagram-v2
    [*] --> Closed State
    
    state Closed State {
        Note1: Normal Operations.<br/>Traffic flows freely downstream.<br/>Track error rate over sliding window.
    }
    
    Closed State --> Open State : Error Rate > Threshold (e.g. > 50% failures)
    
    state Open State {
        Note2: Short-Circuit Active!<br/>Fail Fast Locally (HTTP 503 / Fallback).<br/>Zero network calls sent downstream!
    }
    
    Open State --> HalfOpen State : Reset Timeout Expires (e.g. after 30s)
    
    state HalfOpen State {
        Note3: Probe Mode.<br/>Pass trial requests (e.g. 5 requests).
    }
    
    HalfOpen State --> Closed State : Trial Requests SUCCESS! (Healed)
    HalfOpen State --> Open State : Trial Request FAILS! (Still Broken)

Figure 2: Finite state machine transitions governing Closed, Open, and Half-Open states.

State Definitions and Behaviors

Trip Threshold Mathematics: Sliding Window Failure Rate

Circuit breakers evaluate failure thresholds over rolling time windows (e.g. last 100 requests or last 60 seconds) rather than static counters:

$$E_{\text{rate}} = \frac{N_{\text{failures}} + N_{\text{timeouts}}}{N_{\text{total\_requests}}} \times 100\%$$

flowchart LR
  subgraph Rolling Window (100 Requests)
    Success["80 Successes"]
    Failure["15 Network Exceptions"]
    Timeout["5 Connection Timeouts"]
  end
  
  Calc["Failure Rate = (15 + 5) / 100 = 20%"]
  Check{Failure Rate > 50%?}
  
  Success --> Calc
  Failure --> Calc
  Timeout --> Calc
  Calc --> Check
  Check -->|NO| StayClosed[Remain CLOSED]

Figure 3: Sliding window error rate evaluation.

Defensive Triad: Timeouts, Bulkheads, and Circuit Breakers

A resilient system combines three distinct resilience patterns to form a complete defense against cascading failures:

flowchart TD
  Req[Incoming API Request] --> Bulkhead[1. Bulkhead Isolation Thread Pool]
  Bulkhead --> Timeout[2. Strict Network Timeout Header (e.g. 500ms)]
  Timeout --> CB[3. Circuit Breaker State Machine]
  CB --> Downstream[Downstream Dependency]

Figure 4: The 3-tier defensive layer combining Bulkheads, Timeouts, and Circuit Breakers.

1. Network Timeouts

2. Bulkheads

3. Circuit Breakers

Out-of-Process Circuit Breaking via Service Mesh (Envoy / Istio)

Passive vs Active Health Checking

Complete Worked Example: Production Go Circuit Breaker State Machine

Let's inspect a complete, thread-safe Go implementation of a Circuit Breaker for the TrafficLab platform (trafficlab.com).

package main
import (
	"errors"
	"fmt"
	"sync"
	"time"
)
type State int
const (
	StateClosed State = iota
	StateOpen
	StateHalfOpen
)
var (
	ErrCircuitOpen = errors.New("CIRCUIT_BREAKER_OPEN: Request short-circuited locally")
)
type CircuitBreaker struct {
	mu           sync.Mutex
	state        State
	failureRate  float64       // Trip threshold (e.g. 0.50 for 50%)
	resetTimeout time.Duration // Time to stay in Open before HalfOpen (e.g. 10s)
	
	requests     int64
	failures     int64
	lastStateChange time.Time
}
func NewCircuitBreaker(threshold float64, resetTimeout time.Duration) *CircuitBreaker {
	return &CircuitBreaker{
		state:        StateClosed,
		failureRate:  threshold,
		resetTimeout: resetTimeout,
		lastStateChange: time.Now(),
	}
}
func (cb *CircuitBreaker) Execute(req func() error, fallback func() error) error {
	cb.mu.Lock()
// Check if Open state has expired -> Transition to HalfOpen
	if cb.state == StateOpen {
		if time.Since(cb.lastStateChange) > cb.resetTimeout {
			cb.state = StateHalfOpen
			cb.requests = 0
			cb.failures = 0
			cb.lastStateChange = time.Now()
			fmt.Println("[CIRCUIT BREAKER] Reset timeout expired. Transitioning to HALF-OPEN (Probe Mode).")
		} else {
			cb.mu.Unlock()
			if fallback != nil {
				return fallback()
			}
			return ErrCircuitOpen
		}
	}
cb.mu.Unlock()
// Execute actual network call
	err := req()
cb.mu.Lock()
	defer cb.mu.Unlock()
cb.requests++
	if err != nil {
		cb.failures++
	}
// State transition evaluation
	if cb.state == StateClosed {
		if cb.requests >= 10 { // Minimum request volume before evaluation
			rate := float64(cb.failures) / float64(cb.requests)
			if rate >= cb.failureRate {
				cb.state = StateOpen
				cb.lastStateChange = time.Now()
				fmt.Printf("[CIRCUIT BREAKER] Failure rate %.2f%% exceeded threshold. TRIP OPEN!\n", rate*100)
			}
		}
	} else if cb.state == StateHalfOpen {
		if err == nil {
			cb.state = StateClosed
			cb.requests = 0
			cb.failures = 0
			cb.lastStateChange = time.Now()
			fmt.Println("[CIRCUIT BREAKER] Probe request SUCCESS! Circuit HEALED -> CLOSED.")
		} else {
			cb.state = StateOpen
			cb.lastStateChange = time.Now()
			fmt.Println("[CIRCUIT BREAKER] Probe request FAILED! Re-tripping -> OPEN.")
		}
	}
return err
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Circuit breakers stop cascading failures: Short-circuit failing downstream calls locally to prevent thread pool starvation across upstream services.
2. Master the 3 states: Closed (normal traffic), Open (short-circuited fail-fast), and Half-Open (trial probe testing recovery).
3. Combine Timeouts, Bulkheads, and Breakers: Timeouts bound latency, Bulkheads isolate pools, and Breakers stop bad calls.
4. Enforce minimum request thresholds: Require a minimum request volume (e.g. 20 requests) before evaluating sliding window error percentages.
5. Always provide Fallback Handlers: Return cached data, default values, or clean error responses when the circuit breaker is open.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• TransferService calls FraudCheckService over HTTP.
• FraudCheckService response times spike to 45 seconds during database indexing.

1. Formulate the Circuit Breaker configuration (Failure Rate Threshold, Reset Timeout, Min Request Volume) for the FraudCheckService client.
2. Design a graceful Fallback strategy for money transfers when the FraudCheckService circuit breaker trips OPEN.

Interactive Self-Assessment

Blocking calls on a slow dependency consume all available worker threads in the upstream pool, preventing the service from serving any traffic.

It automatically converts relational database tables into read-only CSV files.

It forces public DNS servers to un-register the company's domain name.

It causes physical hardware CPU clock drift on primary database servers.

It permits a limited number of trial probe requests downstream to test whether the dependency has recovered.

It permanently blocks 100% of all future network calls until the server is rebooted.

It rewrites the application source code files on disk to remove the network call.

It clears all local browser TLS encryption certificate stores.

What to Learn Next

• Backpressure — Flow Control Between Producers and Consumers: Explore reactive stream flow control to prevent buffer overflow.
• Tail Latency & Load Shedding: Discover emergency load shedding techniques for handling extreme traffic spikes.
• Fault Tolerance — Redundancy & Self-Healing Systems: Learn how self-healing architectures maintain uptime across distributed clusters.

Track: Reliability and Operations

Previous: Checksums & Data Integrity

Next: Disaster Recovery — RPO, RTO, and Backups That Work

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab