system-design · advanced
Tail Latency and Load Shedding
Quick answer
Tail latency is the slow fringe (p99/p999) users feel during retries and fan-out. Load shedding deliberately rejects or degrades work when overloaded so the system stays usable for the rest.
Why this matters
- Fan-out multiplies p99 into user-visible stalls.
- Retries amplify overload (retry storms).
- Protecting the core journey beats equal suffering for all traffic.
Learning objectives
- Measure p99/p999, not only averages. 2. Explain fan-out amplification. 3. Apply timeouts, concurrency limits, and shedding. 4. Prefer useful degradation over total collapse.
Explain like I am 5
If the ice-cream line is endless, stop taking new orders so people already waiting still get served.
Mental model
flowchart TD
Load --> Queue
Queue -->|healthy| Serve
Queue -->|overload| Shed[Shed / degrade]
Shed --> Protect[Protect critical path]
Core concepts
Averages lie
Mean latency hides the bad 1%.Fan-out
One request waiting on 20 backends sees the max of tails.Timeouts and deadlines
Propagate deadlines; cancel useless work.Admission control
Limit concurrency/queue depth; fail fast when full.Critical vs optional
Shed recommendations before checkout authorization.Worked example
Search p99 explodes under bot traffic. Add per-IP limits + shed nonlogged-in heavy facets; checkout path untouched; p99 recovers.
Trade-offs
| Never shed | Shed aggressively |
|---|---|
| Meltdown | Angry users on edge features |
Failure modes
| Mode | Mitigation |
|---|---|
| Infinite queues | Bounded queues + 503 |
| Retry storms | Jittered backoff + budget |
| Only avg dashboards | Publish p99 SLIs |
Interview mode
Skeleton: "I design for tails—deadlines, bounded queues, and intentional shedding of noncritical work under overload."
Knowledge check
Keep the system useful by rejecting or degrading excess work
Maximize queue length forever
Delete all monitoring
Always retry immediately without limits
By Shubham Jain