system-design · intermediate

Caching 101 — Memory Offloading and Latency Reduction

The Central Question

Consider an e-commerce platform running on the CacheLab platform (cachelab.com) processing 50,000 requests per second:

• A popular product details page receives 10,000 read queries per second (GET /products/10492).
• Each request queries a relational database (PostgreSQL), executing 4 SQL joins across products, inventory, pricing_rules, and reviews.
• Query execution takes 45 milliseconds on disk, driving database CPU to 98% saturation.

A cache is a temporary high-speed data storage layer (typically operating in volatile RAM) that stores previously computed results so subsequent requests can serve the payload in sub-milliseconds without querying the primary source of truth.

This lesson answers one central question: How do engineers model Cache Hit Ratios (H), calculate Average Memory Access Time (AMAT), implement Cache-Aside access patterns, and manage TTL expiration to protect database infrastructure while maintaining data freshness?

The Hardware Memory Pyramid: RAM vs. Disk Physics

Caching takes advantage of computer hardware physics: RAM is orders of magnitude faster than persistent SSD storage, but carries a higher cost per gigabyte and is volatile.

flowchart TB
  subgraph Memory Hierarchy Speed vs Capacity Pyramid
    L1["1. CPU L1/L2 Cache (0.5 - 1 ns) | Bytes"]
    RAM["2. Main RAM / Redis Cache (0.1 - 1 ms) | Gigabytes"]
    SSD["3. Local NVMe SSD Disk (2 - 10 ms) | Terabytes"]
    RemoteDB["4. Distributed Database Cluster (15 - 50 ms) | Petabytes"]
  end
  
  L1 --> RAM --> SSD --> RemoteDB

Figure 1: The computer hardware memory pyramid illustrating speed, latency, and capacity trade-offs.

Hardware Latency Comparison Table

Core Mechanics: Cache Hits, Cache Misses, and Hit Ratio

When an application queries a cache layer, two outcomes occur:

flowchart TD
  App[Application Request] --> CacheCheck{Is Key in Cache?}
  
  CacheCheck -->|YES: Cache Hit| HitPath["Cache Hit (Sub-ms)<br/>Return Cached Data Immediately!<br/>Latency: 1.5 ms"]
  CacheCheck -->|NO: Cache Miss| MissPath["Cache Miss<br/>Fetch from Database (45 ms)<br/>Write to Cache & Return"]
  
  MissPath --> DB[(Primary Database)]
  DB --> WriteCache[Write Payload to Cache with TTL]

Figure 2: Flowchart depicting Cache Hit vs Cache Miss execution branches.

1. Cache Hit

2. Cache Miss

3. Cache Hit Ratio ($H$)

$$H = \frac{\text{Total Cache Hits}}{\text{Total Cache Hits} + \text{Total Cache Misses}}$$

Mathematical Formulation: Average Memory Access Time (AMAT)

The overall average response latency experienced by an application is governed by the Average Memory Access Time (AMAT) equation:

$$\text{AMAT} = T_{\text{hit}} + (1 - H) \times M_{\text{penalty}}$$

Where:

• $T_{\text{hit}}$ = Time required to query the cache (e.g. $1.5\text{ms}$).
• $H$ = Cache Hit Ratio (e.g. $0.95 = 95\%$).
• $(1 - H)$ = Cache Miss Ratio (e.g. $0.05 = 5\%$).
• $M_{\text{penalty}}$ = Database query latency penalty (e.g. $45\text{ms}$).

Concrete Calculation Example

$$\text{AMAT} = 1.5\text{ms} + (1 - 0.95) \times 45\text{ms} = 1.5\text{ms} + (0.05 \times 45\text{ms}) = 1.5\text{ms} + 2.25\text{ms} = 3.75\text{ms}$$

Compare this to an un-cached database system where average latency is $45\text{ms}$:

• Latency reduction: Reduced from $45\text{ms}$ down to $3.75\text{ms}$ ($91.6\%$ latency improvement).
• Database CPU offload: Reduced database load from $10,000\text{ RPS}$ down to $500\text{ RPS}$ ($95\%$ query reduction).

Principles of Locality: Temporal vs. Spatial Locality

Caching works because computer data access patterns exhibit two predictable behavioral properties:

flowchart TD
  Locality[Principles of Locality] --> Temporal[1. Temporal Locality]
  Locality --> Spatial[2. Spatial Locality]
  
  Temporal --> TDesc["If data is accessed ONCE, it is likely to be accessed AGAIN soon.<br/>Example: Trending breaking news article or hot product page."]
  Spatial --> SDesc["If data item A is accessed, nearby items A+1, A+2 are likely to be accessed soon.<br/>Example: Fetching user profile + profile settings + avatar."]

Figure 3: Comparison of Temporal vs Spatial Locality principles.

Access Pattern: The Cache-Aside (Lazy Loading) Workflow

The most ubiquitous caching pattern in distributed microservices is Cache-Aside:

sequenceDiagram
    autonumber
    actor Client as Application Client
    participant Cache as Redis In-Memory Cache
    participant DB as PostgreSQL Database
    
    Client->>Cache: 1. GET product:10492
    alt Cache Hit (95% of traffic)
        Cache-->>Client: 2. Return JSON Payload (1.5ms)
    else Cache Miss (5% of traffic)
        Cache-->>Client: 3. Return Key Not Found (Nil)
        Client->>DB: 4. SELECT * FROM products WHERE id=10492
        DB-->>Client: 5. Return SQL Record (45ms)
        Client->>Cache: 6. SETEX product:10492 3600 JSON_Payload (TTL: 1 Hour)
        Client-->>Client: 7. Process Request
    end

Figure 4: Sequence diagram of Cache-Aside lazy loading workflow.

Complete Worked Example: Go Thread-Safe In-Memory Cache with TTL

Let's inspect a complete Go implementation of a thread-safe in-memory cache featuring TTL expiration for the CacheLab platform (cachelab.com).

package main
import (
	"context"
	"fmt"
	"sync"
	"time"
)
type cacheItem struct {
	value      interface{}
	expiration time.Time
}
type MemoryCache struct {
	mu    sync.RWMutex
	items map[string]cacheItem
}
func NewMemoryCache(cleanupInterval time.Duration) *MemoryCache {
	c := &MemoryCache{
		items: make(map[string]cacheItem),
	}
go c.startJanitor(cleanupInterval)
	return c
}
func (c *MemoryCache) Set(key string, value interface{}, ttl time.Duration) {
	c.mu.Lock()
	defer c.mu.Unlock()
c.items[key] = cacheItem{
		value:      value,
		expiration: time.Now().Add(ttl),
	}
}
func (c *MemoryCache) Get(key string) (interface{}, bool) {
	c.mu.RLock()
	defer c.mu.RUnlock()
item, exists := c.items[key]
	if !exists {
		return nil, false
	}
// Check TTL Expiration
	if time.Now().After(item.expiration) {
		return nil, false
	}
return item.value, true
}
func (c *MemoryCache) startJanitor(interval time.Duration) {
	ticker := time.NewTicker(interval)
	for range ticker.C {
		c.mu.Lock()
		now := time.Now()
		for k, v := range c.items {
			if now.After(v.expiration) {
				delete(c.items, k)
			}
		}
		c.mu.Unlock()
	}
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Caching offloads RAM over disk: RAM reads take sub-milliseconds ($\sim 0.1-1.5\text{ms}$) compared to disk/network queries ($\sim 15-50\text{ms}$).
2. Hit Ratio ($H$) governs AMAT latency: Higher hit ratios drastically lower Average Memory Access Time ($\text{AMAT} = T_{\text{hit}} + (1-H) \times M$).
3. Exploit Temporal and Spatial Locality: Cache data accessed recently and data likely to be accessed together.
4. Always set explicit Time-To-Live (TTL): Prevent memory exhaustion by setting TTL expiration bounds on all cached keys.
5. Pre-warm cold caches before shifting traffic: Prevent cold boot database crashes by pre-loading hot keys into new cache instances.

Cold Cache Mitigation & Pre-Warming

1. Synthetic warm-up scripts query top 1,000 most frequently accessed keys from database replicas.
2. The results are pre-populated into the cache layer before registering the node with load balancers.
3. Live production traffic is shifted only after the Cache Hit Ratio exceeds $90\%$ in pre-production health checks.

Multi-Tiered Caching (L1 In-Memory + L2 Redis)

• L1 Cache (In-Process Memory): Stored inside application pod memory (e.g. Go sync.Map or Guava Cache) for microsecond access ($T_{\text{hit}} < 50\mu\text{s}$), bypassing network serialization overhead.
• L2 Cache (Distributed Redis): Shared across all application pods for centralized key consistency ($T_{\text{hit}} \approx 1.5\text{ms}$).

Cache Stampede Early Expiration (XFetch)

Practice Scenario and Self-Assessment

Architecture Scenario

• A breaking news article receives 50,000 RPS.
• Database query takes 30ms; Redis cache read takes 1ms.

1. Calculate the Average Memory Access Time (AMAT) if the Cache Hit Ratio is $98\%$.
2. Formulate the Cache-Aside TTL strategy for breaking news articles vs archival articles.

Interactive Self-Assessment

A 98% hit ratio reduces database query load from 20% of traffic down to 2%, achieving a 10x reduction in database CPU queries.

A 98% hit ratio automatically converts relational database schemas into NoSQL tables.

A 98% hit ratio revokes client HTTPS TLS encryption certificates.

A 98% hit ratio doubles the physical hardware clock speed of CPU cores.

The cache map grows monotonically over time, exhausting server RAM and triggering Out-Of-Memory (OOM) process kills.

The cache server automatically changes public DNS nameserver records.

The cache server converts SQL database tables into un-indexed CSV files.

Un-bounded caches cause physical electrical damage to network router cables.

What to Learn Next

• Caching Strategies — Read-Through, Write-Through, Write-Back: Explore write-heavy caching workflows.
• Cache Eviction Policies — LRU, LFU, FIFO, ARC: Master memory eviction algorithms.
• Distributed Caching & Redis Cluster: Learn consistent hashing and Redis cluster partitioning.

Track: Data, Storage and Messaging

Previous: Cache Stampede — When Expiry Melts the Database

Next: Caching Strategies — Aside, Through, Behind, and Refresh-Ahead

Series: Caching

1. Caching 101 — Memory Offloading and Latency Reduction (this guide)
2. Caching Strategies — Aside, Through, Behind, and Refresh-Ahead
3. Cache Eviction Policies — LRU, LFU, TTL, and Friends
4. Distributed Caching — Sharding and High-Availability Clusters
5. Cache Stampede — When Expiry Melts the Database
6. Stale Cache After Write — When Your Own Update Disappears
7. Content Delivery Networks (CDN) — Edge Acceleration and Caching
8. Bloom Filters — Probabilistic Set Membership at Scale

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab