system-design · intermediate

Design a Rate Limiter — Interview Walkthrough

Start here

A rate limiter enforces policies like:

“User 42 may call at most 100 requests per minute.”

It protects APIs from abuse, bugs, and noisy neighbors while keeping the system fair.

This interview classic tests whether you can clarify rules, pick an algorithm, place the component in an architecture, and scale it with shared state.

What you will learn

Gather functional and non-functional requirements.
Compare token bucket, leaky bucket, fixed and sliding windows.
Place limiters at gateway vs service.
Design distributed counters with Redis-style stores.
Return correct HTTP semantics (429, headers).
Discuss failure modes when the limiter store is down.

What you should know first

Topic	Why
Rate limiting algorithms	Algorithm mechanics
API gateway	Common placement
Answering framework	Interview structure

Words you need before we begin

Term	Plain English
Quota	Allowed amount in a period.
Token bucket	Tokens refill over time; requests spend tokens.
Fixed window	Count resets on clock boundaries.
Sliding window	Smoother rolling period approximation.
429	HTTP Too Many Requests.
Hard vs soft limit	Reject vs delay/throttle.
Distributed limiter	Shared state across many servers.

Simple story: tickets for a ride

Riders get tickets over time. No ticket, no ride. Saving tickets allows a short burst. Staff may also smooth entry so people trickle in.

Requirements to clarify

Who is limited: IP, user id, API key, tenant?
Which routes: all, or expensive ones only?
Limits: average rate and burst?
Consistency: approximate OK?
Where enforced: edge, service, both?
Response: 429 body, Retry-After, headers?

Example numbers

10k requests/s globally
Per user 100/min with burst 20
Millisecond overhead budget on allow path

High-level placement

flowchart LR
  Client --> GW[API gateway + limiter]
  GW -->|allow| Svc[Service]
  GW -->|deny 429| Client
  GW --> Store[(Redis counters)]

Edge placement stops bad traffic early. Service placement knows richer identity. Many systems do both (coarse edge + fine service).

Algorithm choice in interviews

Algorithm	Say this
Token bucket	Good default: average rate + burst
Fixed window	Simple but boundary double-spend risk
Sliding window log	Accurate, more memory
Sliding window counter	Practical approximation
Leaky bucket	Smooths to constant rate; may queue

Pick token bucket or sliding window counter and justify.

Distributed design steps

Key design: rate:user:42:/search or similar.
Atomic ops: Lua script or MULTI/INCR with TTL.
Return remaining quota to client when possible.
Shard Redis if needed; accept small inaccuracy if using local+global hybrid.
Fail policy: fail open (risk overload) vs fail closed (safer for fragile deps)—state choice.

Complete worked example

Policy

Authenticated free tier: 60 requests/minute, burst 10, token bucket.

Redis sketch

Store tokens and last refresh timestamp per key.
On request: refill based on elapsed time, clamp to burst, try spend 1.
If insufficient → 429 + Retry-After: 1.

Path

Gateway middleware runs limiter before routing. Health checks excluded. Admin routes separate higher limits.

Failure

Redis blip: for public marketing site maybe fail open with local emergency cap; for payments API fail closed.

Interview scoring signals

Clarified dimensions (who/what/where)
Named algorithm trade-offs
Distributed consistency discussion
Client UX for 429
Not only “use Redis”

Failure modes

Mode	Impact	Mitigation
Fixed window edge burst	2× rate	Sliding/token bucket
Per-IP limits on NAT	Punish many users	Prefer user/API key
Unlimited retries on 429	Storm	Client backoff; temporary bans
Limiter store down	Open or closed risk	Explicit policy + local caps
One key for whole company	Internal noisy neighbor	Tenant + key hierarchy

Trade-offs

Choice	Benefit	Cost
Central Redis	Global fairness	Extra hop, dependency
Local-only limits	Fast	Weak global guarantee
Strict accuracy	Fairness	Complexity/cost
Approximate	Speed/scale	Occasional over-allow

Common misunderstandings

“Rate limit equals security.” One layer only.
“429 means stop forever.” Temporary; honor retry guidance.
“Same limit for all routes.” Login/search often stricter.
“More Redis nodes always keep exact global counts.” Need careful atomic design.
“Token bucket forbids bursts.” Capacity is the burst.

Check your understanding

What two numbers define a token bucket policy?
Why place a limiter at the gateway?
What is a fixed-window boundary problem?
Fail open vs fail closed when Redis is down?
Why exclude health checks?

Practice

Design keys for user + route limits.
Write a 429 response example with headers.
Compare edge-only vs service-only enforcement.
Estimate Redis QPS for 10k RPS with one INCR each.
Role-play a 25-minute design interview.

Deeper production notes

Cost-based tokens

Charge more tokens for heavy endpoints. Document costs so SDKs can budget.

Hierarchy of limits

Global emergency brake + per-tenant + per-user. Prevents one dimension from being gamed.

Observability

Metrics: allowed, denied, store latency, top offenders (careful with PII).

Additional teaching scenarios

Scenario A — peak load day

Traffic multiplies by ten. Mark which failure modes appear first and the first mitigation for each.

Scenario B — mixed versions

Half the fleet runs an old build. Which assumptions break? Prefer one deploy window of compatibility.

Scenario C — five-sentence teach-back

Explain the core idea without acronyms using only the simple story and worked example.

Scenario D — metrics and alerts

List three metrics and one alert that track user impact or a scarce resource.

Scenario E — non-goals

Name two problems this technique should not solve.

Scenario F — ownership

Who owns dashboards, code, and pages? Blank means not ready for broad rollout.

Revision summary

Limiters enforce who may call how often.
Choose algorithm, placement, and shared store.
Return clear 429 semantics.
Decide fail open/closed deliberately.
Show trade-offs, not buzzwords, in interviews.

Glossary

Term	Definition
Rate limiter	Component enforcing request rate policies.
Token bucket	Refilling tokens authorizing requests.
Quota key	Counter identity (user/route/window).

Abbreviations and terminology

RPS/QPS — Requests/queries per second
HTTP 429 — Too Many Requests
NAT — Network Address Translation

What to learn next

Extra teaching notes for first-time builders

Write the single bottleneck you are protecting before picking tools. Name the signal that tells you the design is working for users, not only that internal counters move. When reviewing a change related to this lesson, ask what happens when the component is slow for ten minutes, down entirely, or running twice. Prefer small explicit failure modes that operators can understand at 3 a.m.

Document ownership for dashboards, code, and pages. Undocumented mechanisms become folklore and then outages. Prefer designs that tolerate mixed versions for at least one deploy window so rollouts do not require perfect global simultaneity.

Napkin math helps: estimate peak rate, multiply by payload size, and ask whether the design still holds when a dependency is at half capacity. If the answer depends on luck, add bounds, backpressure, or shedding before production traffic arrives.

Extra teaching notes for first-time builders

FAQ from first-time learners

Q: Is API gateway enough?
A: Great start; some limits need service-level context.

Q: Token bucket or sliding window?
A: Either is defensible—explain burst needs.

Q: Local memory limiter?
A: OK for single instance; weak for horizontal scale.

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab