system-design · intermediate

Scalability — Vertical, Horizontal, and Elastic Growth

The Central Question

Consider a web application running on the CoreLab platform (corelab.com):

• During steady-state operations, a single virtual machine (4 vCPUs, 16 GB RAM) handles 100 HTTP requests per second with an average response time of 35 milliseconds.
• During a major product launch, incoming user traffic surges to 10,000 requests per second. Application CPU hits 100%, RAM exhausts, TCP connection queues fill, and response times spike to 30 seconds before the service crashes with 504 Gateway Timeouts.

Scalability is the structural property of an architectural design that enables a system to process increased workloads predictably by adding compute, memory, disk, or network resources.

This lesson answers one central question: How do engineers model system bottlenecks using Amdahl's Law, design stateless compute pools, and progress sequentially through the scaling ladder to handle 100x traffic growth without exponential cost or latency degradation?

Defining Scalability: Load, Capacity, and Amdahl's Law

To evaluate system scalability, engineers define three fundamental operational concepts:

flowchart LR
  subgraph Offered Load Vector
    L[Traffic Spike: 10,000 req/sec]
  end
  subgraph System Boundary
    C[Max Node Capacity: 2,000 req/sec]
    B[System Bottleneck: DB Write Lock Contention]
  end
  subgraph Operational Result
    L -->|Exceeds Max Capacity| Overflow[Queue Saturation & Cascading Outage]
  end
  
  C -.->|Constrained by| B

Figure 1: Relationship between offered load vectors, maximum system capacity, and component bottlenecks.

1. Load Vectors

• Request Load: HTTP QPS or API calls per second (e.g. 10,000 req/sec).
• Data Load: Total database table size or active record volume (e.g. 500 million rows).
• Concurrent Load: Active open TCP sockets or concurrent WebSocket user channels (e.g. 100,000 connections).

2. Capacity & Bottlenecks

• Capacity: The maximum workload a system processes while satisfying its Service Level Objectives (SLOs), such as $P95 \text{ Latency} < 100\text{ms}$.
• Bottleneck: The specific hardware or software component (CPU, RAM, disk IOPS, network bandwidth, or database lock contention) that reaches 100% saturation first, capping overall system throughput.

3. Amdahl's Law & Speedup Limits

$$\text{Speedup}(S) = \frac{1}{P_{\text{serial}} + \frac{1 - P_{\text{serial}}}{S}}$$

gantt
    title Amdahl's Law Execution Timeline (Serial Bottleneck vs Parallel Tasks)
    dateFormat  ss
    axisFormat %S
    section Non-Parallel Serial Task
    Database Global Row Lock (20%) :crit, s1, 00, 02
    section Parallelizable Tasks
    App Worker 1 Compute (80%) :active, p1, 02, 10
    App Worker 2 Compute (80%) :active, p2, 02, 10
    App Worker 3 Compute (80%) :active, p3, 02, 10

Figure 2: Amdahl's Law illustrating how a 20% serial lock bottleneck caps total system speedup at 5x regardless of node count.

If 20% of an application's execution path requires a global database lock ($P_{\text{serial}} = 0.20$), adding 1,000 server nodes yields a maximum speedup of:

$$\text{Speedup}(\infty) = \frac{1}{0.20 + 0} = 5\times$$

No amount of added compute hardware can overcome a serial architectural bottleneck.

Vertical Scaling (Scale Up) vs. Horizontal Scaling (Scale Out)

Engineers expand system capacity using two primary dimensional vectors:

flowchart TB
  subgraph Vertical Scale Up: Single Node Expansion
    V1["Small Server: 4 vCPU / 16 GB RAM"] -->|Hardware Upgrade| V2["Large Server: 64 vCPU / 256 GB RAM"]
  end
subgraph Horizontal Scale Out: Parallel Node Pool
    H1["Load Balancer"] --> NodeA["App Node 1 (4 vCPU)"]
    H1 --> NodeB["App Node 2 (4 vCPU)"]
    H1 --> NodeC["App Node 3 (4 vCPU)"]
    H1 --> NodeD["App Node 4 (4 vCPU)"]
  end

Figure 3: Comparing vertical hardware upgrades against horizontal node pool expansion.

1. Vertical Scaling (Scale Up)

• Pros: Requires zero code changes; maintains 100% single-node ACID transaction guarantees.
• Cons: Subject to hard cloud hardware instance ceilings; creates a single failure domain (SPOF); incurs non-linear enterprise hardware pricing cliffs.

2. Horizontal Scaling (Scale Out)

• Pros: Provides virtually unlimited capacity expansion; eliminates single points of failure; offers linear cloud infrastructure cost scaling.
• Cons: Requires stateless compute tier design; introduces network latency and distributed consistency trade-offs.

The Critical Divider: Stateless vs. Stateful Architecture

An application tier cannot scale out horizontally unless its compute instances are 100% stateless:

flowchart TB
  subgraph Anti-Pattern: Un-scalable Stateful App Tier
    C1[Client A] -->|Session Stored in Node Memory| N1[App Node 1]
    C1 -.->|Next Request Routed to Node 2| N2[App Node 2]
    N2 --x|Session Not Found!| Err[User Logged Out / HTTP 401]
  end
subgraph Production Pattern: Scalable Stateless App Tier
    C2[Client B] -->|Request with Signed JWT Header| N3[App Node 3]
    C2 -->|Next Request with Signed JWT Header| N4[App Node 4]
    N4 -->|Validate JWT Signature in RAM| Success[Request Processed Successfully]
  end

Figure 4: Comparing stateful local memory dependencies with stateless token architecture.

1. Stateless Compute Tier

• Session Management: Sessions are stored in cryptographically signed tokens (JWTs) presented by the client or in an external shared cache (Redis).
• File Storage: Static assets and file uploads are written directly to distributed object storage (AWS S3) rather than local server drives.

2. Stateful Storage Tier

Auto-Scaling and Elasticity: Headroom Buffers & Provisioning Lag

Elasticity is the operational capability of an architecture to automatically provision compute resources during load surges and decommission them when traffic subsides.

gantt
    title Auto-Scaling Elasticity Timeline (Traffic Spike vs Provisioning Lag)
    dateFormat  HH:mm
    axisFormat %H:%M
    section Traffic Demand
    Baseline Traffic (1,000 req/sec) :done, t1, 00:00, 04:00
    Flash Traffic Spike (8,000 req/sec) :crit, t2, 04:00, 08:00
    Cool-down Traffic (1,500 req/sec) :done, t3, 08:00, 12:00
    section Compute Pool Scale
    4 Nodes Active (Buffer 40%) :active, n1, 00:00, 04:30
    Auto-Scale Trigger & Provisioning Lag :crit, n2, 04:30, 05:00
    24 Nodes Active (Capacity Restored) :active, n3, 05:00, 08:30
    Scale-In Cooldown (Reduce to 6 Nodes) :done, n4, 08:30, 12:00

Figure 5: Auto-scaling curve illustrating alarm evaluation delay, container boot lag, and headroom buffers.

Managing Provisioning Lag

1. Metric alarm evaluation window: 1 to 3 minutes.
2. Cloud virtual machine provisioning: 1 to 2 minutes.
3. Container boot and readiness probe checks: 30 to 60 seconds.

To prevent request queue overflow during provisioning lag, production systems maintain a Headroom Buffer—running baseline compute pools at 50% to 60% average CPU utilization so existing nodes comfortably absorb sudden load spikes while new nodes boot up.

Complete Worked Example: Go Stateless JWT Session Validator

Let's inspect a production Go HTTP middleware implementation for the CoreLab platform (corelab.com) that enables 100% stateless horizontal scaling by validating signed JWT tokens in memory without querying a database or session store.

package main
import (
	"context"
	"fmt"
	"net/http"
	"strings"
	"time"
"github.com/golang-jwt/jwt/v5"
)
type StatelessAuthMiddleware struct {
	jwtSecret []byte
}
type UserClaims struct {
	UserID   string json:"user_id"
	Role     string json:"role"
	TenantID string json:"tenant_id"
	jwt.RegisteredClaims
}
func NewStatelessAuth(secret string) *StatelessAuthMiddleware {
	return &StatelessAuthMiddleware{jwtSecret: []byte(secret)}
}
func (m StatelessAuthMiddleware) Authenticate(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r http.Request) {
		authHeader := r.Header.Get("Authorization")
		if authHeader == "" || !strings.HasPrefix(authHeader, "Bearer ") {
			http.Error(w, "Missing Authorization Header", http.StatusUnauthorized)
			return
		}
tokenString := strings.TrimPrefix(authHeader, "Bearer ")
		claims := &UserClaims{}
// Validate JWT signature and expiration 100% in RAM (Zero DB Seeks!)
		token, err := jwt.ParseWithClaims(tokenString, claims, func(token jwt.Token) (interface{}, error) {
			if _, ok := token.Method.(jwt.SigningMethodHMAC); !ok {
				return nil, fmt.Errorf("unexpected signing method: %v", token.Header["alg"])
			}
			return m.jwtSecret, nil
		})
if err != nil || !token.Valid {
			http.Error(w, "Invalid or Expired Token", http.StatusUnauthorized)
			return
		}
// Inject validated claims into request context for downstream handlers
		ctx := context.WithValue(r.Context(), "user_claims", claims)
		next.ServeHTTP(w, r.WithContext(ctx))
	})
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Amdahl's Law limits speedup: Serial execution paths (global database locks) cap total system speedup regardless of how many nodes are added.
2. Stateless compute is mandatory for Scale Out: Move session state to JWT tokens or shared Redis caches so any app node can handle any request.
3. Scale data in progressive stages: Advance through indexes $\rightarrow$ vertical scaling $\rightarrow$ read replicas $\rightarrow$ caching $\rightarrow$ horizontal sharding.
4. Account for provisioning lag: Booting new nodes takes 3 to 6 minutes. Maintain a 40% headroom buffer to absorb traffic spikes safely.
5. Never scale the un-constrained tier: Verify the true bottleneck (CPU, RAM, disk IOPS, DB locks) before adding hardware.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• Currently, 1 monolithic app server handles user authentication, image binary uploads to local disk /var/uploads, and synchronous CPU-intensive image resizing.
• During peak hours, image resizing drives CPU to 100%, blocking user login requests.

1. Identify 3 architectural choices in this initial design that prevent horizontal scaling.
2. Formulate a 3-step redesign plan to transform this monolith into a horizontally scalable system.

Interactive Self-Assessment

10x maximum theoretical speedup.

1,000x linear speedup.

1x speedup (zero throughput increase).

Infinite speedup.

It allows load balancers to route any request to any healthy instance without losing session context.

Stateless servers require zero memory and zero CPU resources to operate.

Statelessness eliminates the need for database storage systems entirely.

Stateless servers automatically compile code into hardware assembly instructions.

What to Learn Next

• Latency vs Throughput — Performance Measurement and Queuing Math: Master queuing theory and percentile latency metrics.
• System Availability — Nines, Redundancy, and Downtime Budgets: Learn how to design high-availability fault domains.
• Stateful vs Stateless — Architectural Trade-offs and Decision Frameworks: Deep dive into shared state stores and JWT token design.

Track: Software Design and Architecture

Previous: Push vs Pull — Who Initiates Data Movement

Next: Serverless Architecture — Managed Compute on Demand

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab