system-design · beginner

Stale Cache After Write — When Your Own Update Disappears

Start here

You click Save. The UI says success. You refresh — and the old value is back.

That experience often means:

The database (or primary store) accepted your write, but a cache (or lagging copy) still held the previous value and served it on the next read.

This is a read-your-writes failure from the user’s point of view: I just wrote; I should see my write.

You should care because users interpret this as data loss. Support tickets skyrocket. Engineers waste hours “debugging the frontend” when the real bug is cache invalidation ordering.

What you will learn

1. Define the stale-after-write problem in plain English.
2. Separate user-visible success from cache coherence.
3. Walk the classic cache-aside fill-vs-delete race.
4. List other causes: forgotten invalidation, multi-key caches, replica lag.
5. Apply fixes: delete-after-commit, versioning, short TTL, read-through primary, sticky sessions for the writer.
6. Work a profile-edit example.
7. Know what to log and test.
8. Avoid “just clear all caches” as the only strategy.

What you should know first

Words you need before we begin

Simple story: the whiteboard and the photocopy

The whiteboard is the database. The photocopy on the wall is the cache.

1. You update the whiteboard: “Meeting at 4pm.”
2. You forget to throw away the photocopy that still says 3pm.
3. A colleague reads the photocopy and goes at 3pm.

1. Someone copies the old 3pm to a new photocopy while you are erasing.
2. You delete the old photocopy.
3. The new photocopy of 3pm is pinned up after your delete.
4. Everyone sees 3pm again.

Where the analogy stops: computers do this thousands of times per second across many machines.

The problem in a request timeline

User steps

1. PUT /profile → 200 OK with new bio.
2. GET /profile immediately → old bio.

What may have happened

• App wrote Postgres successfully.
• App failed to DEL profile:123 (bug, timeout, wrong key).
• Or app deleted, but another request refilled from a stale read.
• Or GET hit a read replica that was 200 ms behind.
• Or a CDN edge still had the old JSON.

Step-by-step explanation

Step 1 — Decide the promise

For each API, document:

Does the writer need read-your-writes within N milliseconds?

• Account settings: usually yes.
• Global analytics dashboard: often no.
• Public product description: “within a minute” may be OK.

Step 2 — Forgotten invalidation (simplest bug)

Cache-aside write path missing step 2:

1. UPDATE products SET price=…
2. ~~DEL product:{id}~~ ← forgotten
3. Readers keep hitting old cache until TTL.

Step 3 — The classic race (detailed)

Two requests, same key:

Reader R (miss path)
Writer W (update path)

Bad interleaving:

1. R misses cache.
2. R reads DB → value OLD (write not committed yet) or timing variants.
3. W commits NEW to DB.
4. W deletes cache key.
5. R sets cache to OLD.
6. Future readers see OLD until TTL.

Mitigations:

• Keep TTLs short as a backstop.
• Store version in DB and cache; never fill if version is older.
• Use locks around read-fill for hot keys (costly).
• Prefer delete over set-on-write for complex objects; combine with compare-and-set versions.
• Delay populate slightly (hacky) or re-check DB version before SET.

Step 4 — Multi-key and derived caches

You update user row but cache:

• user:1
• feed:home:1
• search:autocomplete:jo

Fix: document dependency graph; invalidate all derived keys; or avoid caching hard-to-invalidate derivatives; or use event-driven invalidation with clear ownership.

Step 5 — Replica lag looks like stale cache

Even with perfect Redis logic:

1. Write primary.
2. Read replica for GET (load balancing).
3. Replica not caught up → old value.

• Read-your-writes: route recent writers to primary for a short window.
• Synchronous replication (latency cost).
• Session sticky to primary after write.
• Return written entity in the write response so UI does not immediately need GET.

Step 6 — CDN and browser caches

For public HTTP responses:

• Cache-Control may allow edges and browsers to keep old JSON/HTML.
• After publish, you must purge CDN or use versioned URLs (app.v3.js).

Step 7 — Practical repair toolkit

Step 8 — Testing the bug on purpose

Add tests or chaos:

• Concurrent read miss + write for same key
• Forced replica delay
• Invalidation timeout injection

Visual mental model

Happy invalidate

flowchart LR
  W[Write DB NEW] --> D[Delete cache]
  D --> M[Next read misses]
  M --> L[Load NEW from DB]
  L --> F[Fill cache NEW]

Learning question: What breaks if Delete is skipped?

Caption: Correct cache-aside write path.

Poison fill race

flowchart TB
  R[Reader loads OLD] --> S[Writer commits NEW + deletes]
  S --> P[Reader populates OLD]
  P --> X[Cache poisoned]

Learning question: Which step should check a version before SET?

Caption: Late populate overwrites a correct delete.

Complete worked example: edit display name

Flow

1. User submits name “Ada”.
2. API updates users.name in Postgres transaction.
3. On commit success: DEL user:9:public.
4. API returns { "name": "Ada" } to the client.
5. Client updates local state from response (no immediate GET required).
6. Other devices may see old name until miss/TTL — acceptable if documented.
7. If another tab GETs within milliseconds, single-flight + version reduces races.

Bug that shipped

Invalidation used key user:9 but readers used user:9:public. Silent mismatch → 15-minute TTL staleness.

Lesson: key naming is part of the contract; centralize key builders.

How it works in production

Logging

On write:

• user id, entity version, cache keys deleted, delete success/failure

• cache hit/miss, value version, DB version sample

Metrics

• Invalidation failure rate
• “Write then read mismatch” synthetic canary
• Replica lag
• CDN purge latency

Product messaging

If eventual visibility is expected cross-device, say “It may take a minute to show everywhere” rather than “Saved” with an immediate global promise you cannot keep.

Failure modes

Trade-offs

Compare with related concepts

Common misunderstandings

1. “HTTP 200 on write means the next GET cannot be stale.”

1. “TTL 24h is fine if we invalidate.”

1. “Only Redis causes this.”

1. “Returning the write body is cheating.”

1. “FLUSHALL in production fixes coherence.”

Check your understanding

A cache fill/invalidation race

A DNS CNAME loop only

Filesystem corruption by definition

Perfect linearizability success

Update the UI from the write response (and invalidate correctly for others)

Never invalidate any keys

Always read a random replica immediately with no lag controls

Disable HTTPS

Practice

Trace this bug report: “I changed my address; the confirmation page shows the new address, but My Account shows the old one for ~2 minutes.”

1. List three cache layers that could be involved.
2. Write a sequence diagram for a fill race.
3. Propose key names and invalidation points.
4. Add one automated test idea.
5. Choose max acceptable staleness for other devices vs the writer.

Revision summary

1. Stale-after-write = truth updated, read path not.
2. Common causes: missed invalidation, races, replica lag, CDN.
3. Delete-after-commit; unify key names.
4. Versions stop poison fills.
5. Serve write response to the writer.
6. Test concurrency; monitor invalidation failures.
7. Document RYW promises per feature.

Glossary

Abbreviations and terminology

What to learn next

Primary next lesson: Content Delivery Network (CDN)

Revisit cache stampede when origin load is the pain instead of wrongness.

Track: Data, Storage and Messaging

Previous: Read-Through vs. Write-Through Cache — Who Updates the Cache?

Series: Caching

1. Caching 101 — Memory Offloading and Latency Reduction
2. Caching Strategies — Aside, Through, Behind, and Refresh-Ahead
3. Cache Eviction Policies — LRU, LFU, TTL, and Friends
4. Distributed Caching — Sharding and High-Availability Clusters
5. Cache Stampede — When Expiry Melts the Database
6. Stale Cache After Write — When Your Own Update Disappears (this guide)
7. Content Delivery Networks (CDN) — Edge Acceleration and Caching
8. Bloom Filters — Probabilistic Set Membership at Scale

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab