system-design · intermediate

Distributed Rate Limiting — Shared Quotas Across Many Pods

The Central Question

Consider a high-throughput API gateway cluster running on the TrafficLab platform (trafficlab.com):

• The edge tier deploys 50 API Gateway Pods behind an AWS Application Load Balancer to process 50,000 requests per second.
• An enterprise client has an agreed quota of 1,000 requests per minute.

In total, the client executes 50,000 requests per minute—exceeding their contract limit by $5,000\%$ because local pods have zero visibility into requests handled by peer instances!

To enforce unified rate limits across multi-node clusters, systems deploy Distributed Rate Limiting.

A Distributed Rate Limiter coordinates request counts across multiple gateway instances using a high-speed, centralized key-value store (such as Redis or Memcached) to maintain an authoritative, global quota state per client.

This lesson answers one central question: How do engineering teams design Distributed Rate Limiters using centralized Redis Lua scripts, eliminate concurrency race conditions, mitigate Redis network round-trip overhead using local batching, and handle Redis cluster outages via Fail-Open resilience policies?

The Distributed State Problem: Local Counters vs. Centralized State

Enforcing rate limits in a distributed environment requires shifting state from pod memory to a centralized memory layer:

flowchart TD
  subgraph Flawed: Isolated Local Memory Counters
    GW1[API Gateway Pod 1<br/>Local Count: 990/1000]
    GW2[API Gateway Pod 2<br/>Local Count: 980/1000]
    GW3[API Gateway Pod 3<br/>Local Count: 995/1000]
    Note1["Flaw: Client gets 3,000 total requests (3x limit!)"]
  end
subgraph Correct: Centralized Redis Shared State
    GW4[API Gateway Pod 1] -->|Atomic Lua Eval| Redis[(Centralized Redis Cluster<br/>Global Key: 'rate:usr_42'<br/>Count: 995/1000)]
    GW5[API Gateway Pod 2] -->|Atomic Lua Eval| Redis
    GW6[API Gateway Pod 3] -->|Atomic Lua Eval| Redis
    Note2["Success: Unified 1,000 request limit enforced globally!"]
  end

Figure 1: Contrast between flawed local pod counters and centralized Redis global rate enforcement.

Concurrency Race Conditions and Redis Lua Scripting

A naive implementation of a distributed rate limiter issues three sequential commands to Redis:

# Anti-Pattern: Non-Atomic Distributed Rate Limiter
def is_allowed(client_id):
    current_tokens = redis.get(f"rate:{client_id}") # 1. Network Read
    if current_tokens > 0:
        redis.set(f"rate:{client_id}", current_tokens - 1) # 2. Network Write
        return True
    return False

The Race Condition Flaw

1. All 10 pods issue redis.get() simultaneously.
2. All 10 pods read current_tokens = 1.
3. All 10 pods issue redis.set() with current_tokens - 1 = 0 and allow the request.
4. 10 requests were allowed when only 1 token was available, causing severe over-allocation under concurrent load.

Solution: Atomic Redis Lua Script Execution

sequenceDiagram
    autonumber
    actor Client as API Client
    participant Pod as Gateway Pod Node
    participant Redis as Redis Server Engine
    
    Client->>Pod: 1. POST /v1/payments (Header: X-API-Key: key_901)
    
    rect rgb(240, 248, 255)
        Note over Pod,Redis: Single Atomic Round-Trip
        Pod->>Redis: 2. EVALSHA <lua_script_hash> 1 "rate:key_901" 10 1.0 (Capacity=10, Refill=1/s)
        Note over Redis: Redis Atomic Lua Execution:<br/>• Compute Elapsed Time since last_updated<br/>• Add Refilled Tokens (min capacity)<br/>• If tokens >= 1: decrement & return 1<br/>• Else: return 0
        Redis-->>Pod: 3. Return Array [Allowed=1, Remaining=9, ResetSeconds=1]
    end
    
    Pod-->>Client: 4. HTTP 200 OK (X-RateLimit-Remaining: 9)

Figure 2: Sequence diagram illustrating atomic Lua script evaluation in Redis.

Production Production Redis Lua Script: Token Bucket

Let's examine the production Redis Lua script used across TrafficLab (trafficlab.com):

-- Production Redis Token Bucket Lua Script
-- KEYS[1]: Rate limit key (e.g. "rate_limit:user_8901")
-- ARGV[1]: Bucket Capacity (e.g. 100)
-- ARGV[2]: Refill Rate per second (e.g. 10.0)
-- ARGV[3]: Current Unix Timestamp in seconds (e.g. 1771239000.125)
-- ARGV[4]: Requested Tokens (e.g. 1)
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])
-- 1. Fetch stored state from Redis Hash
local data = redis.call("HMGET", key, "tokens", "last_updated")
local tokens = tonumber(data[1])
local last_updated = tonumber(data[2])
if tokens == nil then
    -- Initial state: bucket full
    tokens = capacity
    last_updated = now
else
    -- Compute refill since last request
    local delta = math.max(0, now - last_updated)
    tokens = math.min(capacity, tokens + (delta * refill_rate))
    last_updated = now
end
-- 2. Check if enough tokens exist
if tokens >= requested then
    tokens = tokens - requested
    redis.call("HMSET", key, "tokens", tokens, "last_updated", last_updated)
    redis.call("EXPIRE", key, math.ceil(capacity / refill_rate))
    return {1, math.floor(tokens)} -- Allowed = 1
else
    redis.call("HMSET", key, "tokens", tokens, "last_updated", last_updated)
    return {0, math.floor(tokens)} -- Denied = 0
end

Reducing Redis Network Overhead: Local Batching & Asynchronous Sync

While Redis latency is sub-millisecond ($0.5\text{ms}$), adding a synchronous Redis call to every single HTTP request at 50,000 RPS introduces severe network overhead:

• $50,000\text{ requests/sec} \times 1\text{ Redis call} = 50,000\text{ Redis QPS}$.
• Network switch latency spikes and Redis CPU load surges.

flowchart TD
  subgraph Gateway Pod Local Memory
    Req1[Request 1] --> LocalLimiter[Local Token Bucket Buffer]
    Req2[Request 2] --> LocalLimiter
    Req3[Request 3] --> LocalLimiter
    LocalLimiter -->|Allow Instantly!| HTTP[Return HTTP 200 OK]
  end
subgraph Asynchronous Batch Sync (Every 500ms)
    LocalLimiter -->|Batch Deduct 50 Tokens| SyncWorker[Async Background Sync Worker]
    SyncWorker -->|Single Pipeline Call| Redis[(Centralized Redis Cluster)]
  end

Figure 3: Local batching architecture allowing instant pod-level checks and async Redis syncing.

How Local Batching Operates

1. Each API Gateway pod reserves a local batch quota of tokens (e.g. 50 tokens) from Redis.
2. The pod evaluates requests locally in $O(1)$ memory without making network calls.
3. Every 500ms, a background worker syncs consumed tokens back to Redis in a single pipelined operation.
4. Trade-off: Slightly looser rate limit accuracy (up to $5\%$ over-allocation during rapid bursts) in exchange for $95\%$ reduction in Redis network calls.

Resiliency Policies: Fail-Open vs. Fail-Closed

What happens when the centralized Redis cluster suffers a major network partition or node failure?

API Gateways must choose between two failure policies:

flowchart TD
  RedisFail[Redis Connection Failed / Timeout Exceeded!] --> PolicyCheck{Configured Resiliency Policy}
  
  PolicyCheck -->|Fail-Open (Default)| Open["ALLOW Request to Pass Through<br/>• Log Security Warning<br/>• Prioritize System Availability<br/>• Recommended for Public Web APIs"]
  PolicyCheck -->|Fail-Closed| Closed["REJECT Request with HTTP 429 / 503<br/>• Protect Backend Databases<br/>• Prioritize Security & Cost Control<br/>• Recommended for Expensive AI/LLM Inferences"]

Figure 4: Comparing Fail-Open and Fail-Closed fallback policies during Redis outages.

Policy Selection Rules

• Fail-Open (Recommended for Most APIs): If Redis is unreachable, the gateway logs a warning and allows requests through. It is better to temporarily tolerate slightly un-throttled traffic than to cause a $100\%$ global API outage for all paying users.
• Fail-Closed (Recommended for High-Cost APIs): Used for endpoints that trigger expensive downstream operations (e.g. OpenAI GPT-4 API calls costing $\$0.10$ per request or physical credit card authorizations).

Distributed Sliding Window Log in Redis Sorted Sets

1. Each request adds a timestamp member (score = timestamp, member = UUID) via ZADD.
2. The script drops entries older than $(t_{\text{now}} - W)$ using ZREMRANGEBYSCORE.
3. ZCARD counts remaining requests in the rolling window.
4. If count <= limit: allow request; else: reject request with HTTP 429.

Multi-Region Redis Rate Limiting Strategies

Complete Worked Example: Production Go Distributed Rate Limiter Client

Let's inspect a complete Go implementation of a distributed rate limiter client connecting to Redis for the TrafficLab platform (trafficlab.com).

package main
import (
	"context"
	"fmt"
	"time"
"github.com/redis/go-redis/v9"
)
const TokenBucketScript = `
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])
local data = redis.call("HMGET", key, "tokens", "last_updated")
local tokens = tonumber(data[1])
local last_updated = tonumber(data[2])
if tokens == nil then
    tokens = capacity
    last_updated = now
else
    local delta = math.max(0, now - last_updated)
    tokens = math.min(capacity, tokens + (delta * refill_rate))
    last_updated = now
end
if tokens >= requested then
    tokens = tokens - requested
    redis.call("HMSET", key, "tokens", tokens, "last_updated", last_updated)
    redis.call("EXPIRE", key, math.ceil(capacity / refill_rate))
    return {1, math.floor(tokens)}
else
    redis.call("HMSET", key, "tokens", tokens, "last_updated", last_updated)
    return {0, math.floor(tokens)}
end
`
type DistributedLimiter struct {
	rdb        *redis.Client
	scriptHash string
}
func NewDistributedLimiter(rdb redis.Client) (DistributedLimiter, error) {
	hash, err := rdb.ScriptLoad(context.Background(), TokenBucketScript).Result()
	if err != nil {
		return nil, err
	}
	return &DistributedLimiter{rdb: rdb, scriptHash: hash}, nil
}
func (dl *DistributedLimiter) Allow(ctx context.Context, clientKey string, capacity, refillRate float64) (bool, int, error) {
	now := float64(time.Now().UnixNano()) / 1e9
	
	res, err := dl.rdb.EvalSha(ctx, dl.scriptHash, []string{"rate:" + clientKey}, capacity, refillRate, now, 1).Result()
	if err != nil {
		// Fail-Open Resilience Strategy on Redis Error!
		fmt.Printf("[FAIL-OPEN WARNING] Redis Error: %v | Allowing Traffic\n", err)
		return true, 1, nil
	}
results := res.([]interface{})
	allowed := results[0].(int64) == 1
	remaining := int(results[1].(int64))
return allowed, remaining, nil
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Local counters cannot enforce cluster-wide limits: Multi-node gateway pools require a centralized store (Redis) to maintain authoritative client quota state.
2. Always execute rate limit math in Redis Lua scripts: Atomic Lua script execution prevents concurrency race conditions across multi-instance gateway pods.
3. Use Local Batching to scale past 100,000 RPS: Buffer token deductions locally in pod memory and sync asynchronously to Redis to reduce network overhead by 95%.
4. Choose Fail-Open for availability: Fall back to allowing traffic through when Redis experiences an outage, unless protecting high-cost LLM/payment endpoints.
5. Always set TTLs on Redis rate limit keys: Execute EXPIRE in your Lua scripts to prevent stale client keys from exhausting Redis RAM.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• Prompt generation endpoints (POST /v1/chat/completions) cost $\$0.05$ per call.
• The system runs 100 Kubernetes pods across 3 cloud regions.

1. Select the resiliency policy (Fail-Open vs Fail-Closed) for the prompt generation endpoint and justify your choice based on operational cost risks.
2. Formulate how your system avoids Redis Hot-Key bottlenecks when 1,000,000 free-tier users make API requests simultaneously.

Interactive Self-Assessment

Redis executes Lua scripts atomically in a single thread, ensuring no other operations can intervene between token checks and updates.

It converts TCP socket connections into un-encrypted UDP packets.

It forces client web browsers to execute rate limiting code in JavaScript.

It replaces Redis in-memory storage with relational SQL tables.

When protecting high-cost downstream operations (such as expensive LLM inferences or paid credit card APIs) where un-throttled traffic causes severe financial loss.

When building a free public weather forecasting API that prioritizes high availability.

When domain nameservers experience high DNS resolution latency.

When upgrading physical RAM hardware modules on gateway servers.

What to Learn Next

• Circuit Breakers & Cascading Failure Control: Learn how to isolate failing downstream microservices.
• Backpressure — Flow Control Between Producers and Consumers: Explore reactive stream flow control to prevent buffer exhaustion.
• Tail Latency & Load Shedding: Master emergency server self-preservation during extreme traffic overload.

Track: Software Design and Architecture

Previous: API Design — Clear Contracts Clients Can Trust

Next: Rate Limiter Design — Case Study

Series: Rate Limiting

1. Rate Limiting Algorithms — Token Bucket, Windows, and Bursts
2. Rate Limiter Design — Case Study
3. Distributed Rate Limiting — Shared Quotas Across Many Pods (this guide)
4. Circuit Breakers & Rate Limiting Together

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab