system-design · beginner

Load Balancing — Algorithms and Layers

The Central Question

Consider a web application whose domain api.checkoutlab.com resolves to a single IP address 203.0.113.10. If 50,000 users attempt to check out simultaneously, all 50,000 TCP connections concentrate on that single server. The CPU hits 100%, memory exhausts, connection queues fill, and the server crashes. The single IP address becomes a Single Point of Failure (SPOF).

To scale horizontally, an application must distribute incoming traffic across multiple redundant backend instances.

A load balancer is a specialized network component that acts as a reverse proxy, accepting client traffic at a single public entry point and distributing work across a pool of backend servers based on active health status and load algorithms.

This lesson answers one central question: How should incoming client connections and HTTP requests be distributed across a pool of healthy backend instances to maximize throughput, minimize latency, and maintain high availability?

The Reverse Proxy Architecture

A load balancer operates as a Reverse Proxy. Unlike a forward proxy (which hides client IP addresses when browsing out to the internet), a reverse proxy sits in front of backend servers to hide internal server infrastructure from public clients.

flowchart LR
  subgraph Public Internet
    C1[Client Browser A]
    C2[Client Browser B]
  end
  subgraph Public Edge
    LB[Load Balancer 203.0.113.10]
  end
  subgraph Private Network Tier
    App1[App Server 172.16.1.10]
    App2[App Server 172.16.1.11]
    App3[App Server 172.16.1.12]
  end
  
  C1 -->|1. HTTP Request| LB
  C2 -->|1. HTTP Request| LB
  LB -->|2a. Forward| App1
  LB -->|2b. Forward| App2
  LB -.->|2c. Health Failed| App3

Figure 1: The reverse proxy architecture distributing incoming edge requests to internal private app nodes.

Traffic Routing Flow

1. Public Client Connection: The client initiates a TCP handshake with the load balancer's public IP address (203.0.113.10) on port 443.
2. Backend Selection: The load balancer evaluates its target pool, filters out unhealthy instances, and applies a routing algorithm to select an active backend instance (e.g. 172.16.1.10).
3. Internal Forwarding: The load balancer forwards the request to the internal private IP address of the chosen backend server.
4. Response Relay: The backend processes the request and sends the response back to the load balancer, which relays it to the public client.

Layer 4 (Transport) vs. Layer 7 (Application) Load Balancing

Load balancers operate at different OSI network stack layers, determining how deeply they inspect network packets.

flowchart TB
  subgraph Layer 4: Transport-Level Balancing
    L4[L4 Balancer] -->|Inspects IP:Port & TCP Flags| L4Route[Forward TCP Stream to Backend]
  end
  subgraph Layer 7: Application-Level Balancing
    L7[L7 Balancer] -->|Decrypts TLS & Parses HTTP Headers| PathRoute{Inspect URL Path}
    PathRoute -->|/checkout| Pool1[Payment App Pool]
    PathRoute -->|/static| Pool2[Static CDN Origin Pool]
  end

Figure 2: Comparing Layer 4 transport stream forwarding against Layer 7 HTTP content inspection.

1. Layer 4 Load Balancing (Transport Layer)

• Mechanism: The balancer forwards raw TCP packets between the client and the backend server.
• Advantages: Extremely high throughput, low CPU overhead, and capability to balance non-HTTP TCP traffic (e.g. PostgreSQL, Redis, gRPC).
• Disadvantages: Cannot inspect HTTP headers, cookies, or URL paths; cannot perform content-based routing.
• Examples: AWS Network Load Balancer (NLB), HAProxy in TCP mode.

2. Layer 7 Load Balancing (Application Layer)

• Mechanism: The balancer acts as an HTTP proxy, making routing decisions based on HTTP content.
• Advantages: Advanced path-based routing (/api/checkout vs /static/images), HTTP header manipulation (X-Forwarded-For), and cookie-based sticky sessions.
• Disadvantages: Higher CPU consumption per request due to TLS decryption and HTTP parsing.
• Examples: Nginx, AWS Application Load Balancer (ALB), Envoy Proxy.

Routing Algorithms: How Balancers Select Backends

When a load balancer receives a request, it uses a deterministic algorithm to select an active backend instance from the target pool:

flowchart TD
  Req[Incoming Client Request] --> Alg{Select Routing Algorithm}
  Alg -->|Round-Robin| RR[Cyclic Target: Server 1 -> Server 2 -> Server 3]
  Alg -->|Least Connections| LC[Least Busy Target: Server with Fewest Active TCP Sockets]
  Alg -->|IP / Consistent Hash| Hash[Hash Ring Target: Hash Client IP to Fixed Server]

Figure 3: Overview of common load balancer target selection algorithms.

1. Round-Robin & Weighted Round-Robin

• Round-Robin: Rotates requests sequentially across the target pool ($S_1 \rightarrow S_2 \rightarrow S_3 \rightarrow S_1$).

• Weighted Round-Robin: Assigns static weights to instances based on hardware capacity. A server with weight 3 receives three times as many requests as a server with weight 1.

2. Least Connections

• Best For: Long-lived connections (WebSockets, database connection pools, file uploads) where request execution durations vary significantly.

3. IP Hash & Consistent Hashing

• IP Hash: Computes a hash of the client's source IP address (hash(client_ip) % N) to map a specific user IP to the same backend server.
• Consistent Hashing: Maps client keys and backend servers onto a logical $360^\circ$ hash ring. When backend nodes are added or removed, only a small fraction of keys are re-mapped ($1/N$), preventing global cache invalidation.

Active Health Probes, Flapping, and Connection Draining

A load balancer is only as effective as its ability to detect and remove unhealthy backends.

stateDiagram-v2
    [*] --> Healthy
    
    Healthy --> Unhealthy : 2 Consecutive Failed Health Probes (6s)
    note right of Healthy
        Instance receives live traffic.
        LB sends GET /healthz every 3s.
    end note
    
    Unhealthy --> Draining : Failure Detected
    note right of Unhealthy
        Removed from active routing table.
        Zero new client requests sent.
    end note
    
    Draining --> Healthy : 3 Consecutive Successful Probes
    note right of Draining
        Existing TCP connections drain.
        Wait for connection_draining_timeout (30s).
    end note

Figure 4: State machine governing backend health checks and connection draining.

1. Active Health Checking

• Probe Target: GET /healthz on HTTP port 8080.
• Interval: Every 3 seconds.
• Unhealthy Threshold: 2 consecutive failed responses (e.g. HTTP 500 or timeout $> 1,000\text{ms}$).
• Recovery Threshold: 3 consecutive successful HTTP 200 OK responses before re-entering the pool.

2. Flapping Prevention

3. Connection Draining (Graceful Deregistration)

1. It stops sending new client requests to the instance immediately.
2. It keeps existing active TCP connections open for a specified timeout (e.g. 30 seconds) to allow in-flight requests to complete cleanly.
3. Once in-flight requests finish or the timeout expires, the socket is closed.

TLS Termination Models

Because Layer 7 load balancing requires reading HTTP content, load balancers manage Transport Layer Security (TLS/HTTPS) encryption using three deployment models:

flowchart LR
  subgraph Model 1: TLS Termination at Edge
    Client1[Client Browser] -->|HTTPS Encrypted| LB1[Load Balancer]
    LB1 -->|Un-encrypted HTTP| App1[Private App Server]
  end
  subgraph Model 2: End-to-End mTLS Re-Encryption
    Client2[Client Browser] -->|HTTPS Encrypted| LB2[Load Balancer]
    LB2 -->|Internal HTTPS mTLS| App2[Private App Server]
  end

Figure 5: Comparing TLS edge termination against internal mTLS re-encryption.

1. TLS Termination (Offloading)

• Benefit: Centralizes SSL/TLS certificate management on the load balancer; reduces application server CPU load by offloading cryptographic handshakes.

1. TLS Pass-Through (Layer 4):

1. End-to-End Re-Encryption (mTLS):

Sticky Sessions (Session Affinity) vs. Stateless Design

When legacy applications store user session data in server local memory rather than a centralized Redis cache, the load balancer must use Sticky Sessions (Session Affinity).

flowchart TB
  subgraph Sticky Session Pinning
    ClientA[Client A] -->|Cookie: LB_STICKY=NODE_1| LB[Load Balancer]
    LB -->|PINNED| Node1[App Node 1]
    ClientB[Client B] -->|Cookie: LB_STICKY=NODE_2| LB
    LB -->|PINNED| Node2[App Node 2]
  end

Figure 6: Cookie-based sticky session pinning tying specific clients to physical backend nodes.

How Sticky Sessions Work

Operational Pitfalls of Sticky Sessions

1. Uneven Load Distribution: If a high-volume client is pinned to Node 1, that node becomes overloaded while adjacent nodes sit idle.
2. Broken Failover: If Node 1 crashes, all users pinned to Node 1 lose their session state and are forcibly logged out when routed to Node 2.
3. Horizontal Autoscaling Obstacle: Auto-scaling cannot decommission Node 1 cleanly without breaking active pinned users.

Complete Worked Example: Path-Based Routing for CheckoutLab

Let's design a Layer 7 load balancer architecture for the CheckoutLab e-commerce service.

Routing Specification

• Domain: https://checkoutlab.com
• https://checkoutlab.com/api/checkout/* $\rightarrow$ Forward to Payment Target Group (High CPU, 8 Nodes).
• https://checkoutlab.com/api/catalog/* $\rightarrow$ Forward to Catalog Read Target Group (High RAM, 4 Nodes).
• https://checkoutlab.com/static/* $\rightarrow$ Forward to Static Asset Storage Pool.

Execution Sequence Under Active Outage

sequenceDiagram
    autonumber
    actor Client as User Browser
    participant LB as L7 Load Balancer (Nginx / ALB)
    participant Pay1 as Payment Node 1 (DEADLOCK)
    participant Pay2 as Payment Node 2 (HEALTHY)
    participant Stat as Static Asset Pool
    
    Client->>LB: GET /static/logo.png
    LB->>Stat: Route to Static Pool
    Stat-->>Client: HTTP 200 OK (5ms)
    
    Client->>LB: POST /api/checkout/pay
    LB->>Pay1: Route to Payment Target Group (Least Connections)
    Note over Pay1: Times out (Health Probe fails 2x)
    Note over LB: LB marks Pay1 UNHEALTHY; triggers Connection Draining
    LB->>Pay2: Automatic Retry Forward to Payment Node 2
    Pay2-->>LB: HTTP 200 OK { orderId: 8812 }
    LB-->>Client: HTTP 200 OK (Total Latency: 110ms)

Figure 7: Path-based routing and automated health check failover on an L7 load balancer.

Load Balancing vs. Related Network Traffic Management Concepts

Failure Modes and Engineering Mitigations

What You Should Remember

1. Load Balancers are Reverse Proxies: They sit in front of private instance pools, acting as a stable public entry point while instances scale or fail.
2. Layer 4 vs Layer 7: Layer 4 balances fast at the TCP level; Layer 7 decrypts HTTP to enable path-based routing (/api vs /static) and header inspection.
3. Match Algorithm to Workload: Use Round-Robin for uniform workloads; use Least Connections for variable request durations (WebSockets, long polling).
4. Health checks must reflect readiness: Shallow health probes lead to black-hole routing. Probes must verify critical database and thread pool health.
5. Prefer Stateless Tiers over Sticky Sessions: Sticky sessions cause uneven load distribution and break graceful failovers. Move session state to Redis.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

1. Web Frontend (Static HTML/JS assets)
2. Payment API (HTTP REST, short execution time)
3. Live Chat API (Persistent WebSockets, long-lived connections)

1. For each of the three services, recommend whether Layer 4 or Layer 7 load balancing is appropriate, and justify your choice.
2. Select the optimal balancing algorithm (Round-Robin vs Least Connections vs IP Hash) for the Live Chat API and explain why.

Interactive Self-Assessment

Layer 7 (Application Layer)

Layer 4 (Transport Layer)

Layer 3 (Network Layer)

Layer 2 (Data Link Layer)

Least Connections tracks active open connections, preventing traffic imbalances when connection durations vary widely.

Round-Robin requires more CPU memory to compute than Least Connections.

Least Connections is the only algorithm that supports encrypted TLS sockets.

Least Connections automatically bypasses database locks during WebSocket handshakes.

What to Learn Next

• Scalability — Vertical, Horizontal, and Elastic Growth: Revisit horizontal compute tier expansion with load balancing in mind.
• Availability — Nines, Error Budgets, and Redundancy: Learn how load balancer health probes protect system availability targets.
• Single Point of Failure: Explore high-availability load balancer pairs (VRRP / floating IPs).

Track: Engineering Foundations

Previous: Leader Election — Picking One Coordinator

Next: Long Polling vs WebSockets — Realtime Over HTTP

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab