system-design · intermediate

Reliability — Correct Results Under Stress

The Central Question

Consider a financial ledger processing API running on the CoreLab platform (corelab.com):

• The API processes 500,000 requests per day. The server returns an HTTP 200 OK for 100% of incoming requests in under 30 milliseconds.
• By traditional reachability metrics, the system appears 100% available. However, during minor network router blips, client HTTP retries cause 1,200 customers to be debited twice on their account balances without creating matching order records.

System reliability is not just about staying online. It is about data correctness, invariant preservation, and trust under stress.

This lesson answers one central question: How do engineers define system invariants, calculate MTBF and MTTR metrics, and design atomic, idempotent execution pipelines that guarantee correct data outputs even when underlying components fail or retry?

Defining Reliability: Correctness, Invariants, and Time

Reliability is the probability that a system performs its specified intended function correctly, preserving business invariants and data integrity under stated operational conditions over a defined time window.

flowchart TD
  Req[Incoming Client Operation] --> Reach{Is Service Reachable?}
  Reach -->|No: HTTP 503 / Timeout| AvailFail[Availability Failure: Un-reachable]
  Reach -->|Yes: HTTP 200 OK| Correct{Are Data Invariants Preserved?}
  Correct -->|No: Double Charge / Data Loss| RelFail[Reliability Failure: Silent Corruption]
  Correct -->|Yes: Valid State Change| Success[System Operational Success]
  
  style AvailFail fill:#f8d7da,stroke:#f5c6cb
  style RelFail fill:#f8d7da,stroke:#f5c6cb
  style Success fill:#d4edda,stroke:#c3e6cb

Figure 1: Decision tree distinguishing availability failures (reachability) from reliability failures (correctness).

The Three Core Pillars of Reliability

1. Intended Function: The explicit behavioral contract promised to users.

1. System Invariants: Rules that must remain true across every state change, regardless of load or component failure.

1. Operational Period: Reliability is evaluated over weeks and months of production traffic, incorporating database failovers, deployment pushes, and traffic spikes.

Availability vs. Reliability: The Critical Distinction

Engineers frequently treat availability and reliability as synonyms, but they measure fundamentally different properties:

flowchart LR
  subgraph High Availability / Low Reliability
    A1[HTTP 200 Returned Instantly] --> A2[Database Writes Skipped / Corrupted]
  end
  subgraph High Reliability / Low Availability
    B1[Scheduled Maintenance Window: Downtime] --> B2[Zero Data Loss / Invariants 100% Intact]
  end

Figure 2: Contrasting high availability with high reliability.

An unavailable system stops work. An unreliable system does wrong work. Silent wrongness destroys customer trust faster than an explicit outage.

Threats to System Reliability

Software reliability is threatened by partial failures, concurrent operations, and distributed network boundaries:

flowchart TB
  subgraph Production Reliability Threats
    T1[Transient Network Drops] --> R1[Client Retries Cause Duplicate Side Effects]
    T2[App Server Mid-Execution Crash] --> R2[Partial State Writes Left in Database]
    T3[Concurrent Request Races] --> R3[Two Threads Overwrite Same Database Record]
    T4[Stale Cache Read] --> R4[Stale Prices Served as Live Data]
  end

Figure 3: Common distributed system mechanics that cause silent reliability failures.

1. Partial Failures and Incomplete Transactions

2. Unsafe Retries (Lack of Idempotency)

3. Concurrency Race Conditions

Measuring Failure and Recovery: MTBF and MTTR

Operations teams track reliability trends using two foundational metrics:

timeline
    title Reliability Metric Timeline (MTBF vs MTTR)
    section Normal Operation
        System Running Correctly : Operational Window (MTBF Starts)
    section Failure Event
        Hardware / Software Outage : Failure Occurs (MTBF Ends)
    section Recovery Phase
        Incident Response & Repair : Restoration Window (MTTR)
    section Resumed Operation
        System Restored to Healthy State : New Operational Window

Figure 4: Visualizing Mean Time Between Failures (MTBF) and Mean Time To Repair (MTTR).

1. Mean Time Between Failures (MTBF)

$$\text{MTBF} = \frac{\text{Total Operational Hours}}{\text{Number of Failure Incidents}}$$

2. Mean Time To Repair (MTTR)

$$\text{MTTR} = \frac{\text{Total Maintenance Downtime Hours}}{\text{Number of Failure Incidents}}$$

Proactive Fault Injection & Chaos Engineering

Continuous Reconciliation & Audit Engines

Complete Worked Example: Go Idempotent Financial Transaction Executor

Let's inspect a production Go idempotent transaction handler for the CoreLab platform (corelab.com) that executes atomic account transfers while enforcing strict idempotency deduplication.

package main
import (
	"context"
	"database/sql"
	"errors"
	"fmt"
	"time"
)
type ReliableTransferService struct {
	db *sql.DB
}
type TransferRequest struct {
	IdempotencyKey string  json:"idempotency_key"
	FromAccount    string  json:"from_account"
	ToAccount      string  json:"to_account"
	Amount         float64 json:"amount"
}
func (s *ReliableTransferService) ExecuteIdempotentTransfer(ctx context.Context, req TransferRequest) (string, error) {
	tx, err := s.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelReadCommitted})
	if err != nil {
		return "", fmt.Errorf("failed to begin transaction: %w", err)
	}
	defer tx.Rollback() // Guarantees cleanup on unexpected panics
// 1. Check Idempotency Key Table with Pessimistic Row Lock
	var existingTxID string
	err = tx.QueryRowContext(ctx, 
		"SELECT tx_id FROM idempotency_ledger WHERE idempotency_key = $1 FOR UPDATE", 
		req.IdempotencyKey).Scan(&existingTxID)
if err == nil {
		// Key exists! Return cached transaction ID cleanly without re-executing transfer
		tx.Commit()
		fmt.Printf("[IDEMPOTENT REUSE] Key %s already processed. Returning cached TxID: %s\n", req.IdempotencyKey, existingTxID)
		return existingTxID, nil
	}
if !errors.Is(err, sql.ErrNoRows) {
		return "", fmt.Errorf("idempotency check error: %w", err)
	}
// 2. Validate Sender Balance and Lock Row
	var balance float64
	err = tx.QueryRowContext(ctx, "SELECT balance FROM accounts WHERE id = $1 FOR UPDATE", req.FromAccount).Scan(&balance)
	if err != nil || balance < req.Amount {
		return "", fmt.Errorf("insufficient balance or account missing: %w", err)
	}
// 3. Execute Atomic Transfer Operations
	_, err = tx.ExecContext(ctx, "UPDATE accounts SET balance = balance - $1 WHERE id = $2", req.Amount, req.FromAccount)
	if err != nil {
		return "", err
	}
_, err = tx.ExecContext(ctx, "UPDATE accounts SET balance = balance + $1 WHERE id = $2", req.Amount, req.ToAccount)
	if err != nil {
		return "", err
	}
// 4. Record Idempotency Key
	generatedTxID := fmt.Sprintf("TX_%d", time.Now().UnixNano())
	_, err = tx.ExecContext(ctx, 
		"INSERT INTO idempotency_ledger (idempotency_key, tx_id, created_at) VALUES ($1, $2, NOW())", 
		req.IdempotencyKey, generatedTxID)
	if err != nil {
		return "", err
	}
// 5. Commit Transaction Atomically
	if err := tx.Commit(); err != nil {
		return "", fmt.Errorf("commit failed: %w", err)
	}
fmt.Printf("[TRANSFER COMPLETED] Successfully transferred $%.2f. TxID: %s\n", req.Amount, generatedTxID)
	return generatedTxID, nil
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Uptime is not Reliability: An application returning HTTP 200 OK while corrupting database records is available, but completely unreliable.
2. State system invariants explicitly: Define exact rules (e.g. Balance >= 0, Unique Email) and enforce them at the database tier using schema constraints and ACID transactions.
3. Design for mandatory retries: Distributed network calls will time out. Every state-mutating API must implement idempotency keys to ensure safe client retries.
4. Prevent partial execution: Group multi-step state mutations into atomic units of work so they either succeed completely or roll back entirely.
5. Optimize both MTBF and MTTR: High reliability requires lengthening the operational time between failures (MTBF) while automating recovery to shorten repair duration (MTTR).

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

1. The customer's credit card is charged $\$30$.
2. A restaurant notification is placed in a message queue.
3. The restaurant's inventory of ingredients is decremented.

1. Explain what happens to system reliability if the application server crashes immediately after Step 1 (credit card charged).
2. Detail how you would use database transactions and idempotency keys to make this 3-step workflow reliable under server crashes and network retries.

Interactive Self-Assessment

It has high availability and low reliability.

It has high reliability and low availability.

It has low latency and low availability.

It is both highly available and highly reliable.

They allow clients to safely retry dropped network calls without triggering duplicate charges.

They compress HTTP payload data to increase network bandwidth.

They automatically fix load balancer health check interval timing.

They convert SQL databases from relational storage to key-value caches.

What to Learn Next

• System Availability — Nines, Redundancy, and Downtime Budgets: Learn how to design high-availability fault domains.
• Scalability — Vertical, Horizontal, and Elastic Growth: Discover how to maintain system performance and reliability under heavy load.
• Stateful vs Stateless — Architectural Trade-offs and Decision Frameworks: Deep dive into shared state stores and JWT token design.

Track: Reliability and Operations

Previous: Heartbeats — Liveness Signals in Distributed Systems

Next: Retry Storms — When Recovery Makes the Outage Worse

Series: Reliability & Failover

1. Availability — Nines, Error Budgets, and Redundancy
2. Reliability — Correct Results Under Stress (this guide)
3. Fault Tolerance — Keep Working When Parts Fail
4. Failover — Switching to a Healthy Spare
5. Multi-Region Failover — Surviving a Region Outage
6. Disaster Recovery — RPO, RTO, and Backups That Work

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab