system-design · intermediate

Availability — Nines, Error Budgets, and Redundancy

The Central Question

Consider an API endpoint running on the CoreLab platform (corelab.com):

• When a customer clicks "Submit Order", they expect the service to process their request immediately. If the checkout API returns an HTTP 503 Service Unavailable error or hangs indefinitely, the application is unavailable for that customer — regardless of how well it performed yesterday.

This lesson answers one central question: How do engineers calculate system availability using MTBF and MTTR, model serial vs parallel availability decay, enforce SRE error budgets, and design multi-AZ architectures that achieve Four Nines (99.99%) availability?

Defining Availability: Time-Based vs Request-Based

Availability is measured across two distinct operational paradigms:

flowchart TD
  subgraph Time-Based Uptime Measurement
    T1[Total Monthly Window: 720 Hours] --> T2[Total Outage Time: 43.2 Minutes]
    T2 --> T3["Availability = 99.9% Uptime"]
  end
subgraph Request-Based Success Measurement
    R1[Total Valid Requests: 5,000,000] --> R2[Failed 5xx Requests: 5,000]
    R2 --> R3["Availability = 99.9% Success Rate"]
  end

Figure 1: Comparing time-based uptime against request-based success metrics.

1. Time-Based Availability (Uptime Ratio)

$$\text{Availability}_{\text{time}} = \frac{\text{MTBF}}{\text{MTBF} + \text{MTTR}} \times 100\%$$

Where:

• MTBF (Mean Time Between Failures): The average operational duration a system runs cleanly between crashes.
• MTTR (Mean Time To Repair): The average duration required to detect a failure, reboot a process, and restore traffic routing.

$$\text{Availability}_{\text{time}} = \frac{720}{720 + 0.72} = \frac{720}{720.72} \approx 99.9\%$$

2. Request-Based Availability (API Success Rate)

$$\text{Availability}_{\text{request}} = \frac{\text{Successful Requests}}{\text{Total Valid Requests}} \times 100\%$$

• If an API receives 5,000,000 valid HTTP requests over a month and 5,000 requests fail with HTTP 5xx server errors:

Request-based metrics reflect true user experience because they scale directly with user volume.

The Language of "Nines" and Downtime Budgets

Engineers state availability targets in "nines" — standard target percentages ranging from two nines ($99\%$) to five nines ($99.999\%$).

gantt
    title Allowed Monthly Downtime Windows Across Nines Targets
    dateFormat  mm:ss
    axisFormat %M:%S
    section 99% (Two Nines)
    7 Hours 12 Mins Downtime :crit, n2, 00:00, 43:12
    section 99.9% (Three Nines)
    43 Minutes 12 Seconds :active, n3, 00:00, 04:19
    section 99.99% (Four Nines)
    4 Minutes 19 Seconds :done, n4, 00:00, 00:25

Figure 2: Monthly allowed downtime windows corresponding to target Nines.

How Serial Dependencies Multiply Unavailability

When a client request must traverse a series of dependent microservice components, total availability is not the average of component availabilities. Serial availability is the product of each component's individual availability.

flowchart LR
  Client[Client Browser] -->|Req| GW[API Gateway: 99.9%]
  GW -->|Req| App[App Server: 99.9%]
  App -->|Req| DB[(Database: 99.9%)]
  
  style GW fill:#fff3cd,stroke:#ffebaa
  style App fill:#fff3cd,stroke:#ffebaa
  style DB fill:#fff3cd,stroke:#ffebaa

Figure 3: A serial request chain where overall availability degrades with every added dependency.

The Serial Availability Formula

$$A_{\text{total}} = \prod_{i=1}^{N} A_i = A_1 \times A_2 \times A_3 \times \dots \times A_N$$

• If an API Gateway ($99.9\%$), an Application Server ($99.9\%$), and a Database ($99.9\%$) are linked in series:

Connecting three "Three Nines" services in series results in a system that delivers only 99.7% availability (~2.1 hours of downtime per month). Every mandatory serial dependency pulls total availability down.

Raising Availability via Parallel Redundancy

$$A_{\text{parallel}} = 1 - \prod_{i=1}^{N} (1 - A_i) = 1 - (1 - A_1)(1 - A_2)$$

• If an application server has an availability of $99\%$ ($0.99$), running two independent instances in parallel yields:

Running two 99% available instances in parallel produces a 99.99% available tier.

Operational SRE Framework: SLI, SLO, and Error Budgets

High-availability engineering relies on three core Site Reliability Engineering (SRE) constructs: SLI, SLO, and Error Budget.

flowchart TB
  subgraph Measurement & Targets
    SLI[SLI: Measured Signal - % HTTP 200s < 300ms]
    SLO[SLO: Targeted Goal - 99.9% over 30 Days]
  end
  subgraph Budget Management
    EB[Error Budget: 0.1% Allowed Failures]
    EB -->|Budget > 0%| Ship[Feature Velocity: Fast Deploys]
    EB -->|Budget Exhausted| Freeze[Deploy Freeze: Focus on Stability]
  end
  
  SLI -->|Evaluated against| SLO
  SLO -->|Derives| EB

Figure 4: Operational control loop connecting SLIs, SLOs, and feature deployment speed.

1. Service Level Indicator (SLI)

• Example: The proportion of POST /checkout requests that return an HTTP 200 OK status code in under 300 milliseconds, measured at the load balancer.

2. Service Level Objective (SLO)

• Example: POST /checkout SLI will be $\ge 99.9\%$ over any 30-day rolling window.

3. Error Budget

$$\text{Error Budget} = 100\% - \text{SLO}$$

• For a $99.9\%$ SLO over 5,000,000 monthly requests, the Error Budget is 5,000 failed requests ($0.1\%$).
• If a bad configuration update causes 4,000 failed requests in a single afternoon, 80% of the monthly error budget is consumed. The engineering team must pause non-essential feature deployments and prioritize infrastructure reliability until the budget recovers.

Complete Worked Example: Go Multi-AZ Active Health Check Checker

Let's inspect a production Go health monitor for the CoreLab platform (corelab.com) that continuously issues active health probes across multi-AZ instance pools to automate fast failover.

package main
import (
	"context"
	"fmt"
	"net/http"
	"sync"
	"time"
)
type InstanceHealth struct {
	URL          string
	IsHealthy    bool
	ConsecFails  int
	LastCheck    time.Time
}
type MultiAZHealthMonitor struct {
	mu           sync.Mutex
	instances    map[string]*InstanceHealth
	maxFailures  int
	checkInterval time.Duration
}
func NewHealthMonitor(urls []string, maxFails int, interval time.Duration) MultiAZHealthMonitor {
	m := &MultiAZHealthMonitor{
		instances:    make(map[string]InstanceHealth),
		maxFailures:  maxFails,
		checkInterval: interval,
	}
	for _, url := range urls {
		m.instances[url] = &InstanceHealth{URL: url, IsHealthy: true}
	}
	return m
}
func (m *MultiAZHealthMonitor) Start(ctx context.Context) {
	ticker := time.NewTicker(m.checkInterval)
	go func() {
		for {
			select {
			case <-ticker.C:
				m.probeAll(ctx)
			case <-ctx.Done():
				ticker.Stop()
				return
			}
		}
	}()
}
func (m *MultiAZHealthMonitor) probeAll(ctx context.Context) {
	m.mu.Lock()
	urls := make([]string, 0, len(m.instances))
	for u := range m.instances {
		urls = append(urls, u)
	}
	m.mu.Unlock()
client := http.Client{Timeout: 2 * time.Second}
for _, url := range urls {
		req, _ := http.NewRequestWithContext(ctx, "GET", url+"/healthz", nil)
		resp, err := client.Do(req)
m.mu.Lock()
		inst := m.instances[url]
		inst.LastCheck = time.Now()
if err != nil || resp.StatusCode != http.StatusOK {
			inst.ConsecFails++
			if inst.ConsecFails >= m.maxFailures && inst.IsHealthy {
				inst.IsHealthy = false
				fmt.Printf("[FAILOVER ALERT] Node %s declared UNHEALTHY (Fails: %d). Removed from LB!\n", url, inst.ConsecFails)
			}
		} else {
			resp.Body.Close()
			if !inst.IsHealthy {
				fmt.Printf("[RECOVERY ALERT] Node %s recovered HEALTHY. Restored to LB!\n", url)
			}
			inst.IsHealthy = true
			inst.ConsecFails = 0
		}
		m.mu.Unlock()
	}
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Availability is a ratio, not a feeling: Measure availability using request-based success ratios ($\frac{\text{Successful Requests}}{\text{Total Requests}}$) rather than simple server pings.
2. Every Nine gets exponentially harder: Four Nines ($99.99\%$) permits only 4.3 minutes of downtime per month. Achieving it requires full automation.
3. Serial dependencies multiply unavailability: $A_{\text{total}} = A_1 \times A_2 \times A_3$. Every mandatory serial hop reduces total system availability.
4. Parallel redundancy restores availability: $A_{\text{parallel}} = 1 - (1 - A_1)(1 - A_2)$. Running independent replicas in parallel masks single-instance crashes.
5. Manage stability with Error Budgets: Your Error Budget ($100\% - \text{SLO}$) dictates how much operational risk you can afford before freezing feature releases.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

1. Authentication Service (Availability: $99.9\%$)
2. Account Ledger API (Availability: $99.95\%$)
3. Fraud Detection Engine (Availability: $99.0\%$)

1. If all three services are executed in a strict serial chain, calculate the maximum possible overall availability for a checkout request.
2. How can you redesign the call to the Fraud Detection Engine so that its 99.0% availability does not degrade the core checkout path?

Interactive Self-Assessment

5,000 failed requests.

500 failed requests.

50,000 failed requests.

50 failed requests.

Overall serial availability is the product of individual component availabilities.

Load balancers automatically average serial availabilities.

Serial dependencies increase total system redundancy.

HTTP status codes reset failure probabilities at each hop.

What to Learn Next

• Reliability — Correct Results Under Stress: Learn how to ensure data correctness and invariant preservation.
• Scalability — Vertical, Horizontal, and Elastic Growth: Explore how to scale instance capacity to maintain availability.
• Load Balancing — Algorithms and Layers: Master traffic distribution and active health probe routing.

Track: Reliability and Operations

Next: Checksums & Data Integrity

Series: Reliability & Failover

1. Availability — Nines, Error Budgets, and Redundancy (this guide)
2. Reliability — Correct Results Under Stress
3. Fault Tolerance — Keep Working When Parts Fail
4. Failover — Switching to a Healthy Spare
5. Multi-Region Failover — Surviving a Region Outage
6. Disaster Recovery — RPO, RTO, and Backups That Work

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab