system-design · intermediate

Latency vs Throughput vs Bandwidth

The Central Question

Consider two web service API endpoints running on the CoreLab platform (corelab.com):

• Endpoint A: Processes 10,000 requests per second. Every single request takes 15 milliseconds to complete.
• Endpoint B: Processes 10,000 requests per second. 9,900 requests take 10 milliseconds, but 100 requests hang for 8,000 milliseconds (8 seconds) waiting for a database row lock.

Mistaking completion rates for user waiting times leads to flawed engineering decisions.

This lesson answers one central question: How do single-operation delay (latency), operational completion rate (throughput), and pipe capacity (bandwidth) differ, and how do Little's Law, queueing saturation curves, and percentile metrics (p95/p99) isolate system performance bottlenecks?

Defining the Core Metrics

Engineers evaluate system performance across three distinct operational dimensions:

flowchart TD
  subgraph Single Request Metric
    L["1. Latency: Delay per Operation<br/>e.g. 35 ms wait time"]
  end
  subgraph System Rate Metric
    T["2. Throughput: Completion Frequency<br/>e.g. 10,000 requests/sec"]
  end
  subgraph Network Pipe Metric
    B["3. Bandwidth: Carrying Capacity<br/>e.g. 10 Gbps channel width"]
  end

Figure 1: Distinguishing single-operation delay (Latency), completion frequency (Throughput), and pipe capacity (Bandwidth).

1. Latency (Duration of One Operation)

• Focus: The waiting time experienced by an individual user or calling microservice.
• Concrete Example: A GET /user/profile API call takes $35\text{ms}$ from client request dispatch to JSON response parsing.

2. Throughput (Completions Per Unit Time)

• Focus: The total operational volume processing capacity of a server or cluster.
• Concrete Example: A database cluster processes 15,000 SQL SELECT queries per second.

3. Bandwidth (Channel Carrying Capacity)

• Focus: The physical data pipe width between network endpoints.
• Concrete Example: A cloud data center fiber link provides $10\text{ Gbps}$ of network throughput capacity.

Sequential Breakdown of Request Latency

Total user latency is not a monolithic number; it is the sum of discrete, sequential processing stages:

timeline
    title Sequential Breakdown of Single Request Latency (Total: 185ms)
    section Network Transit
        DNS Resolution : 15ms
        TCP Handshake & TLS Setup : 40ms
    section Server Execution
        Queue Wait Time : 80ms
        Application CPU Logic : 10ms
        Database Query Execution : 30ms
    section Response Return
        Network Payload Transit : 10ms

Figure 2: Component breakdown of total end-to-end user latency.

$$\text{Latency}_{\text{total}} = T_{\text{DNS}} + T_{\text{TCP/TLS}} + T_{\text{Queue}} + T_{\text{AppCPU}} + T_{\text{Database}} + T_{\text{Transit}}$$

• If a server's CPU processes requests in $10\text{ms}$, but incoming traffic forces requests to wait in a worker thread queue for $80\text{ms}$ before execution begins, total latency is $185\text{ms}$.
• Optimizing code execution speed from $10\text{ms}$ to $2\text{ms}$ reduces user latency by only $4\%$, because queue wait time ($80\text{ms}$) dominates the latency breakdown.

Bandwidth vs. Latency: The Cargo Ship Analogy

Bandwidth and latency operate independently. High bandwidth does not guarantee low latency.

flowchart TB
  subgraph High Bandwidth / High Latency
    Ship[Cargo Ship with 10,000 Hard Drives] -->|Data Payload: 80 Petabytes| Time1[Transit Duration: 12 Days]
  end
  subgraph Low Bandwidth / Low Latency
    Fiber[Fiber Optic Packet Push] -->|Data Payload: 1 Kilobyte| Time2[Transit Duration: 5 Milliseconds]
  end

Figure 3: Comparing high-bandwidth/high-latency physical transport against low-bandwidth/low-latency fiber signals.

If you fill a cargo ship with thousands of 8 TB NVMe drives and sail across the ocean:

• Bandwidth: Enormous (Petabytes of data delivered per day).
• Latency: Terrible (12 days for the first byte to arrive).

Queueing Theory & Saturation: Why Queues Explode Latency

The relationship between system utilization, throughput, and latency is governed by queueing theory.

flowchart LR
  subgraph Load Arrival Rate
    In[Incoming Requests: 950 req/sec]
  end
  subgraph Server Execution
    Worker[Worker Pool: Max 1,000 req/sec]
  end
  subgraph Utilization Impact
    U1["50% Utilization: Queue Empty (Latency ~10ms)"]
    U2["90% Utilization: Queue Saturates (Latency ~90ms)"]
    U3["99% Utilization: Queue Explodes (Latency -> 990ms)"]
  end
  
  In --> Worker

Figure 4: Non-linear latency growth as resource utilization approaches 100%.

1. Little's Law

$$L = \lambda \times W$$

• If an API receives $1,000\text{ req/sec}$ ($\lambda$) and each request takes $0.2\text{ seconds}$ ($W$) to complete:

2. The Saturation Curve ($M/M/1$ Model)

$$\text{Queue Wait} \approx \frac{U}{1 - U} \times \text{Service Time}$$

• At 50% Utilization: $\frac{0.5}{1 - 0.5} = 1\times \text{Service Time}$.
• At 90% Utilization: $\frac{0.9}{1 - 0.9} = 9\times \text{Service Time}$.
• At 99% Utilization: $\frac{0.99}{1 - 0.99} = 99\times \text{Service Time}$.

Statistical Analysis: Why Averages Lie (Percentiles & Tail Latency)

Evaluating system performance using average (mean) latency creates deceptive metrics.

gantt
    title Latency Distribution Histogram (99 Fast Requests vs 1 Slow Request)
    dateFormat  s
    axisFormat %S
    section 99 Requests (Fast Path)
    Fast Requests (40ms Average) :active, f1, 00, 1s
    section 1 Request (Tail Outlier)
    Lock Contention Wait (5,000ms) :crit, s1, 00, 5s

Figure 5: Visualizing how tail latency outliers distort user experience while remaining obscured by averages.

The Flaw of Averages

$$\text{Average Latency} = \frac{(99 \times 40) + 5,000}{100} = \frac{3,960 + 5,000}{100} = 89.6\text{ ms}$$

An average latency of $89.6\text{ms}$ suggests excellent performance. In reality, $1\%$ of users experienced a catastrophic $5\text{-second}$ delay.

Understanding Percentiles (p50, p95, p99)

• p50 (50th Percentile / Median): The latency threshold where 50% of requests are faster and 50% are slower. Represents the typical user experience.
• p95 (95th Percentile): The threshold where 95% of requests are faster and 5% are slower. Represents heavy traffic conditions.
• p99 (99th Percentile / Tail Latency): The threshold where 99% of requests are faster and 1% are slower. Captures worst-case tail latency caused by database lock waits, garbage collection pauses, or cold starts.

Complete Worked Example: Go Moving Percentile Latency Calculator

Let's inspect a production Go thread-safe latency recorder for the CoreLab platform (corelab.com) that tracks latency percentiles (p50, p95, p99) in real time.

package main
import (
	"fmt"
	"sort"
	"sync"
	"time"
)
type LatencyTracker struct {
	mu       sync.Mutex
	samples  []float64 // Stores duration in milliseconds
	maxCount int
}
func NewLatencyTracker(maxSamples int) *LatencyTracker {
	return &LatencyTracker{
		samples:  make([]float64, 0, maxSamples),
		maxCount: maxSamples,
	}
}
func (t *LatencyTracker) Record(duration time.Duration) {
	t.mu.Lock()
	defer t.mu.Unlock()
ms := float64(duration.Microseconds()) / 1000.0
	if len(t.samples) >= t.maxCount {
		t.samples = t.samples[1:] // Sliding window drop oldest
	}
	t.samples = append(t.samples, ms)
}
func (t *LatencyTracker) GetPercentile(p float64) float64 {
	t.mu.Lock()
	defer t.mu.Unlock()
if len(t.samples) == 0 {
		return 0
	}
// Copy and sort samples
	sorted := make([]float64, len(t.samples))
	copy(sorted, t.samples)
	sort.Float64s(sorted)
index := int(p / 100.0 * float64(len(sorted)-1))
	return sorted[index]
}
func main() {
	tracker := NewLatencyTracker(1000)
// Simulate 99 fast requests (10ms to 40ms) and 1 slow request (5000ms)
	for i := 0; i < 99; i++ {
		tracker.Record(time.Duration(10+i%30)  time.Millisecond)
	}
	tracker.Record(5000  time.Millisecond) // Tail outlier
fmt.Printf("[LATENCY TELEMETRY]\n")
	fmt.Printf("p50 (Median): %.2f ms\n", tracker.GetPercentile(50))
	fmt.Printf("p95:          %.2f ms\n", tracker.GetPercentile(95))
	fmt.Printf("p99 (Tail):   %.2f ms\n", tracker.GetPercentile(99))
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Latency, Throughput, and Bandwidth are distinct: Latency is single-operation wait time, throughput is completion frequency, and bandwidth is raw network channel width.
2. Never rely on Average Latency: Averages hide tail outliers. Evaluate user experience using p95 and p99 percentiles.
3. Queues explode at high utilization: Running servers near 100% CPU causes queue wait times to grow non-linearly. Maintain headroom buffers (50-60% target utilization).
4. Little's Law governs active concurrency: $L = \lambda \times W$. Reducing request latency directly decreases concurrent memory load on app servers.
5. Optimize for product context: Interactive APIs require low latency (p99 $< 200\text{ms}$); background processing pipelines require high throughput.

Glossary of Terms

Practice Scenario and Self-Assessment

Performance Scenario

• Throughput: 500 images processed per minute.
• Average Latency: 120 milliseconds.
• p99 Latency: 9,500 milliseconds (9.5 seconds).
• Worker CPU Utilization: 97%.

1. Explain why average latency ($120\text{ms}$) fails to reflect the user experience of the 1% tail.
2. Using queueing theory, explain why worker CPU utilization at 97% is causing the $9.5\text{-second}$ p99 tail latency spike, and propose two architectural changes to reduce p99 latency to $< 500\text{ms}$.

Interactive Self-Assessment

Queue wait time grows non-linearly as resource utilization approaches 100% saturation.

Network bandwidth automatically shrinks when CPU utilization is high.

Average throughput decreases to zero whenever latency increases.

The p50 percentile metric automatically overrides system CPU scheduling.

Average latency obscures severe tail outliers experienced by the slowest 1% of user requests.

p99 latency calculates network bandwidth consumption across all client links.

p99 latency is easier to compute than arithmetic averages on database servers.

p99 latency measures throughput rates rather than user waiting times.

What to Learn Next

• Scalability — Vertical, Horizontal, and Elastic Growth: Learn how to scale instance pools horizontally to maintain low p95 latency.
• System Availability — Nines, Redundancy, and Downtime Budgets: Learn how to design high-availability fault domains.
• Load Balancing — Algorithms and Layers: Discover how reverse proxy load balancers distribute requests to prevent server queue saturation.

Track: Engineering Foundations

Next: Process vs Thread — Isolation and Shared Memory

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab