system-design · beginner

Cache Stampede — When Expiry Melts the Database

Start here

A cache stampede (also called dogpiling or a thundering herd on a cache key) is this failure mode:

A hot piece of data expires (or is missing). Hundreds or thousands of requests arrive at the same moment. Every request misses the cache and independently recomputes or reloads the same value from the database. The origin collapses under duplicate work.

You should care because the cache was supposed to protect the database. A stampede turns a normal expiry into a self-inflicted outage — often right after a deploy, a TTL boundary, or a cache restart.

What you will learn

1. Define stampede / dogpile in plain English.
2. Explain hot keys and synchronized expiry.
3. Walk a timeline of a stampede second by second.
4. Compare fixes: single-flight, locking, probabilistic early refresh, stale-while-revalidate, background refresh.
5. See how TTL jitter reduces synchronized expiry.
6. Work a homepage and product-detail example.
7. Know metrics and failure modes of the fixes themselves.
8. Decide when longer TTL alone is not enough.

What you should know first

Words you need before we begin

Simple story: the shared answer sheet

A classroom of 200 students may copy the answer to question 1 from the board (cache).

When the teacher erases the board (TTL expiry):

• Without coordination, all 200 run to the library (database) for the same book page.
• The library stampede hurts everyone.
• Single-flight: one student fetches; others wait and copy that one answer.
• Early refresh: a monitor rewrites the board before class arrives, so nobody stampedes.

The problem: success that becomes failure

Healthy state

• Key home:v3 lives in Redis for 60 seconds.
• Hit ratio is high.
• Database load is calm.

Stampede moment

At second 60.000, the key expires. In the next 50 ms, 5,000 requests arrive (homepage traffic).

Without protection:

1. 5,000 cache misses
2. 5,000 identical SQL queries or service calls
3. Database CPU spikes
4. Timeouts → retries → even more load
5. Site appears “randomly down” every minute on the minute

Step-by-step explanation

Step 1 — Why hot keys matter

Not all keys stampede equally. A key hit once per minute can expire safely. A key hit 10,000 times per second will amplify a miss into a crisis.

Examples of hot keys:

• Homepage blob
• “Product of the day”
• Global feature-flag document
• Celebrity profile after a viral event

Step 2 — Synchronized expiry makes it worse

If thousands of keys share TTL = 3600 and were warmed together at deploy time, they may expire together → many stampedes or one giant origin spike.

Jitter: set TTL to 3600 + random(0..120) so expiries spread out.

Step 3 — Cold cache is a stampede cousin

After Redis restart or FLUSHALL:

• Every key misses
• Origin sees “everyone at once”

Step 4 — Fix A: single-flight (request coalescing)

Idea: for each key, only one in-flight load is allowed.

1. Miss detected.
2. Try to acquire “load lock” for key.
3. Winner loads origin, fills cache, releases lock.
4. Losers wait (or poll) and then read the filled cache (or share the winner’s result in-process).

Step 5 — Fix B: stale-while-revalidate

Store value + soft expiry + hard expiry:

• Before soft expiry: normal hit.
• Between soft and hard: serve stale, trigger one background refresh.
• After hard: must miss or block.

HTTP CDNs use related ideas (stale-while-revalidate cache directives).

Step 6 — Fix C: probabilistic early recomputation

Before expiry, each request has a small probability to refresh early:

• Most requests still hit.
• A few refresh “just in case.”
• Probability rises as expiry nears (XFetch-style algorithms).

Step 7 — Fix D: background warmers

A worker refreshes known hot keys on a schedule before TTL death.

Pros: predictable.
Cons: must know the hot set; wasteful if traffic disappears; still need jitter.

Step 8 — Fix E: longer TTL alone

Longer TTL reduces stampede frequency but:

• Increases staleness
• Does not fix cold-cache events
• Does not fix invalidation-driven mass deletes

Step 9 — Locks can stampede too

If 5,000 threads wait on one lock and all wake when it releases, you can get a thundering herd on the lock itself. Good designs:

• Only a few waiters
• Others serve stale
• Or sleep with jitter and recheck cache

Visual mental model

Stampede timeline

flowchart TB
  H[Hot key expires] --> M[Many concurrent misses]
  M --> D[Duplicate origin loads]
  D --> O[Origin overload]
  O --> T[Timeouts and retries]
  T --> O

Learning question: Where should you collapse duplicate work?

Caption: Many misses must become one load.

Single-flight

flowchart LR
  R1[Request 1] --> L{Lock key}
  R2[Request 2] --> L
  R3[Request 3] --> L
  L -->|winner| DB[(Origin load)]
  DB --> C[(Fill cache)]
  C --> All[All requests get value]

Learning question: What happens if the winner crashes mid-load?

Caption: Locks need timeouts so waiters are not stuck forever.

Complete worked example: viral product page

Setup

• Key product:42:view
• TTL 30 seconds
• Peak 8,000 rps on that product during a TV mention
• Render costs 40 ms DB + 20 ms CPU

Without protection

Every 30 seconds: ~8,000 identical rebuilds → DB connection pool exhausts → checkout also fails (shared pool).

With protection

1. TTL jitter 30–45 seconds.
2. Single-flight per instance + Redis lock for cross-instance.
3. Soft TTL at 20s: serve stale until 45s while one refresher runs.
4. Alert if origin builds for product:42 exceed N/second.

Outcome

Origin builds drop to roughly one per refresh window; p99 stays flat through the viral spike.

How it works in production

Building blocks

• In-process single-flight maps
• Redis locks / SET key NX EX
• CDN edge stale-while-revalidate
• Sidecar warmers for top-N keys from analytics

Metrics

• Miss rate on hot key prefixes
• Concurrent origin loads per key (ideal near 1)
• Lock wait time
• DB CPU correlated with TTL boundaries (sawtooth pattern)

Ownership

When a stampede hits, know:

• Which key
• Which service owns the loader
• Whether to serve stale or fail closed

Failure modes

Trade-offs

Compare with related concepts

Common misunderstandings

1. “Just increase TTL to one day.”

1. “Stampede only happens at big tech scale.”

1. “Locks make everything slow.”

1. “CDN means no stampede.”

1. “Single-flight fixes correctness bugs.”

Check your understanding

Many concurrent misses reload the same hot key from origin

The cache runs out of disk sectors only

DNS TTL becomes infinite automatically

HTTPS is disabled

It spreads expiry times so fewer keys die at once

It encrypts cache values

It replaces all load balancers

It deletes the need for a database

Practice

Your news homepage key expires every 15 seconds. Traffic is 2,000 rps. Rebuild takes 100 ms and hits 12 queries.

1. Estimate worst-case origin rebuilds per expiry without protection.
2. Design single-flight across 20 app instances.
3. Add soft/hard expiry numbers.
4. List three metrics for a dashboard panel “stampede risk.”
5. Decide: serve 5-second-stale homepage during refresh, or show an error page?

Revision summary

1. Stampede = concurrent hot misses × expensive rebuild.
2. Synchronized TTL and cold cache make it worse.
3. Collapse duplicate loads (single-flight).
4. Prefer serving stale briefly over melting origin.
5. Jitter TTLs; warm carefully.
6. Measure concurrent rebuilds per key, not only hit ratio.

Glossary

Abbreviations and terminology

What to learn next

Primary next lesson: Stale Cache After Write — Read-Your-Writes Failures

Also: CDN for edge-level caching and coalescing behavior.

Track: Data, Storage and Messaging

Previous: Cache Eviction Policies — LRU, LFU, TTL, and Friends

Next: Caching 101 — Memory Offloading and Latency Reduction

Series: Caching

1. Caching 101 — Memory Offloading and Latency Reduction
2. Caching Strategies — Aside, Through, Behind, and Refresh-Ahead
3. Cache Eviction Policies — LRU, LFU, TTL, and Friends
4. Distributed Caching — Sharding and High-Availability Clusters
5. Cache Stampede — When Expiry Melts the Database (this guide)
6. Stale Cache After Write — When Your Own Update Disappears
7. Content Delivery Networks (CDN) — Edge Acceleration and Caching
8. Bloom Filters — Probabilistic Set Membership at Scale

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab