system-design · beginner

Backpressure — Slow Down When the System Is Full

The Central Question

Consider a flash sale event running on the TrafficLab platform (trafficlab.com):

• During normal operations, 100 client frontend instances publish 500 order notification events per second ($\lambda = 500\text{ msg/sec}$) to a background notification worker cluster.
• The worker cluster processes messages at 600 events per second ($\mu = 600\text{ msg/sec}$). Because consumer throughput exceeds producer ingress ($\mu > \lambda$), queues stay empty and notifications send within 50 milliseconds.
• When the flash sale launches, producer traffic spikes 10x to 5,000 events per second ($\lambda = 5,000\text{ msg/sec}$).

1. The queue grows by 4,400 messages every second ($\text{Lag Rate} = \lambda - \mu = 5,000 - 600$).
2. Within 10 minutes, the queue holds 2,640,000 un-processed messages, consuming 12 Gigabytes of RAM.
3. The worker server runs out of memory (OOM), crashes, and causes notification processing delays of 6 hours.

To protect system resources against rate imbalances, distributed architectures apply Backpressure.

Backpressure is a feedback control mechanism where a busy consumer signals upstream producers to slow down, pause, or drop incoming work when incoming message rates exceed downstream processing capacity.

This lesson answers one central question: How do distributed systems propagate backpressure upstream using Bounded Buffers, Pull-Based Polling, HTTP 429/503 Rate Signaling, and Load Shedding to prevent Memory Exhaustion (OOM) and Bufferbloat latency spikes?

The Physics of Imbalance: Unbounded Queues vs. Bounded Buffers

Understanding why backpressure is mandatory maps to consumer vs. producer rates:

flowchart TD
  subgraph Unbounded Queue (Outage Vector)
    P1[Producer: 5,000 msg/sec] -->|Pushes unchecked| UQ[Unbounded RAM Queue]
    UQ -->|Grows infinitely: +4,400 msg/sec| C1[Consumer: 600 msg/sec]
    UQ -.->|Memory Exhaustion| OOM[CRASH: Out Of Memory!]
  end
  
  subgraph Bounded Queue with Backpressure (Self-Preserving)
    P2[Producer: 5,000 msg/sec] -->|1. Push to Bound| BQ[Bounded Queue: Max 10,000]
    BQ -->|2. Process| C2[Consumer: 600 msg/sec]
    BQ --x|3. FULL! Backpressure Signal| P2
    P2 -->|4. Slow Down / Reject 429| Client[Client Browser]
  end

Figure 1: Contrast between unbounded queue memory crashes and bounded queue backpressure protection.

Mathematical Queue Growth Model

$$\text{Lag}(t) = \int_{0}^{t} (\lambda(\tau) - \mu(\tau)) \, d\tau$$

Where:

• $\lambda(t)$ is the incoming producer arrival rate.
• $\mu(t)$ is the downstream consumer processing rate.

• In an Unbounded Queue: Memory consumption grows linearly with time ($M(t) \propto \text{Lag}(t)$), ending in a Process Crash ($OOM$).
• In a Bounded Buffer: Queue depth is capped at $Q_{\text{max}}$. Once $\text{Lag}(t) = Q_{\text{max}}$, the buffer refuses new writes, forcing upstream components to handle the rate imbalance immediately.

Upstream Backpressure Mechanisms

Backpressure propagates upstream across architectural boundaries using four primary techniques:

flowchart LR
  BackpressureMethods[Backpressure Propagation Vectors]
  
  BackpressureMethods --> Bounded[1. Bounded Buffers]
  BackpressureMethods --> Pull[2. Pull-Based Consumption]
  BackpressureMethods --> HTTPSig[3. HTTP 429 / 503 Signaling]
  BackpressureMethods --> TCPWin[4. TCP Windowing Flow Control]
  
  Bounded --> BoundedDesc["Cap queues at Q_max.<br/>Refuse inserts when full."]
  Pull --> PullDesc["Consumer polls when ready.<br/>Natural rate control."]
  HTTPSig --> HTTPDesc["API Gateway returns 429 / 503.<br/>Forces client pause."]
  TCPWin --> TCPDesc["Zero-Window TCP packets.<br/>Freezes sender socket buffer."]

Figure 2: Taxonomy of the four primary backpressure propagation mechanisms.

1. Bounded Buffers & Rejection Policies

2. Pull-Based Consumption (Kafka / SQS)

In a Pull-Based Model (such as Apache Kafka or AWS SQS), consumers explicitly request batch messages (poll(timeout) or ReceiveMessage) only when they have free processing capacity. If a consumer slows down, it simply delays issuing the next poll request. The un-read messages remain safely buffered on physical broker disks, providing automatic backpressure safety.

3. HTTP 429 / 503 Upstream Signaling

• HTTP 429 Too Many Requests: Signals that client request rates exceed allowed quotas. Includes a Retry-After: 30 header instructing the client to pause for 30 seconds.
• HTTP 503 Service Unavailable: Signals that downstream servers are saturated and shedding load.

4. TCP Windowing Flow Control

Reactive Streams & Demand-Based Flow Control

Bufferbloat: The Hidden Danger of Excessive Buffering

A naive response to queue overflow is simply increasing the queue buffer size from 1,000 to 1,000,000 messages.

While a 1,000,000-message buffer prevents memory crash drops, it creates a catastrophic failure mode known as Bufferbloat:

sequenceDiagram
    autonumber
    actor User as Client Request
    participant Q as 1,000,000-Message Queue Buffer
    participant Worker as Consumer Worker Pool
    
    User->>Q: 1. Enqueue Request #999,999
    Note over Q: Request waits behind 999,998 items!
    Note over Q,Worker: Queue Wait Time = 45 Minutes!
    
    Worker->>Q: 2. Dequeue Request #999,999 (45 minutes later!)
    Worker-->>User: 3. Return HTTP 200 OK
    Note over User: User closed browser 44 minutes ago! Work wasted!

Figure 3: Sequence diagram detailing Bufferbloat latency inflation.

Complete Worked Example: Go Channel Backpressure Implementation

Let's inspect the production Go implementation of a bounded channel backpressure worker pool for the TrafficLab platform (trafficlab.com).

package main
import (
	"context"
	"errors"
	"fmt"
	"net/http"
	"sync"
	"time"
)
type Job struct {
	ID      string
	Payload string
}
type BackpressureDispatcher struct {
	jobQueue chan Job // Bounded Channel Buffer
	workers  int
	wg       sync.WaitGroup
}
func NewDispatcher(bufferCapacity int, workerCount int) *BackpressureDispatcher {
	return &BackpressureDispatcher{
		jobQueue: make(chan Job, bufferCapacity), // Capped Buffer Size!
		workers:  workerCount,
	}
}
func (d BackpressureDispatcher) Start(ctx context.Context) {
	for i := 0; i < d.workers; i++ {
		d.wg.Add(1)
		go func(workerID int) {
			defer d.wg.Done()
			for {
				select {
				case <-ctx.Done():
					return
				case job, ok := <-d.jobQueue:
					if !ok {
						return
					}
					// Process Job
					time.Sleep(100  time.Millisecond)
					fmt.Printf("[Worker %d] Completed Job %s\n", workerID, job.ID)
				}
			}
		}(i)
	}
}
// SubmitJob applies Non-Blocking Backpressure using select default
func (d *BackpressureDispatcher) SubmitJob(job Job) error {
	select {
	case d.jobQueue <- job:
		// Job successfully buffered!
		return nil
	default:
		// Queue is FULL! Apply Backpressure (Fail Fast 429)
		return errors.New("BACKPRESSURE_REJECT: Queue buffer full")
	}
}
func HTTPHandler(dispatcher BackpressureDispatcher) http.HandlerFunc {
	return func(w http.ResponseWriter, r http.Request) {
		job := Job{ID: r.URL.Query().Get("id"), Payload: "checkout_event"}
		
		err := dispatcher.SubmitJob(job)
		if err != nil {
			// Signal Backpressure to upstream HTTP client!
			w.Header().Set("Retry-After", "5")
			w.WriteHeader(http.StatusTooManyRequests) // HTTP 429
			w.Write([]byte("System busy: Queue full. Please retry in 5 seconds."))
			return
		}
		
		w.WriteHeader(http.StatusAccepted) // HTTP 202
		w.Write([]byte("Job accepted for processing"))
	}
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Backpressure protects system survival: Backpressure signals upstream producers to slow down when downstream consumers are full, preventing OOM crashes.
2. Bound every buffer: Always cap queues, channels, and thread pools with explicit maximum capacities ($Q_{\text{max}}$).
3. Beware of Bufferbloat: Excessive queue depth increases wait latency to minutes, forcing workers to process stale, timed-out requests.
4. Prefer Pull-Based Consumption: Pull-based brokers (Kafka/SQS) naturally regulate event consumption rates based on consumer availability.
5. Fail fast with HTTP 429 / 503: Return explicit status codes with Retry-After headers to force clients to desynchronize retry traffic.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

Users upload high-resolution images via an API gateway, which places processing tasks into a queue for background thumbnail generation workers.

System constraints:

• API Gateway receives 2,000 image uploads/sec during peak hours.
• Worker fleet can generate a maximum of 400 thumbnails/sec.

1. Select the optimal backpressure strategy (Bounded Buffer Rejection vs Pull-Based Scaling vs Load Shedding) for the image worker queue.
2. Detail how your API Gateway responds to users when the thumbnail processing queue reaches maximum capacity.

Interactive Self-Assessment

mu), the queue depth grows continuously until the process exhausts physical RAM and crashes with an Out-Of-Memory (OOM) error.">The queue depth grows continuously until it exhausts server RAM, crashing the process with an Out-Of-Memory (OOM) error.

Unbounded queues automatically format the primary database hard drive.

Unbounded queues disable HTTPS TLS certificate encryption on client connections.

Unbounded queues double the physical CPU clock speed of consumer servers.

Consumers explicitly poll for new messages only when they have free capacity, leaving excess messages safely buffered on broker disk.

Pull-based consumption automatically deletes all secondary indexes on SQL database tables.

Pull-based consumption replaces client DNS resolvers with static hosts files.

Pull-based consumption converts standard SQL queries into NoSQL JSON documents.

What to Learn Next

• Tail Latency & Load Shedding: Discover emergency load shedding techniques for handling extreme traffic spikes.
• Circuit Breakers and Cascading Failure Control: Learn how to fail fast when dependencies experience outages.
• Rate Limiting Algorithms — Token Bucket, Windows, and Bursts: Revisit client rate limits and token bucket refill formulas.

Track: Data, Storage and Messaging

Previous: Data Contracts and Ownership

Next: CDC vs Dual Writes — Keeping Two Stores in Sync

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab