system-design · intermediate

Message Queues — Hand Work Off Without Blocking the User

The Central Question

Consider an online checkout platform running on the MessageLab platform (messagelab.com) processing 100,000,000 asynchronous tasks per day:

• Incoming user traffic spikes from a baseline of 500 checkout requests per second to a peak of 10,000 requests per second.
• When a customer completes a purchase, the system must process a PDF invoice document and submit a fraud risk check payload to an external partner API.
• Generating the PDF invoice and issuing the third-party fraud check takes 1,200 milliseconds per order.
• The external fraud API can handle a maximum throughput of 500 requests per second before returning HTTP 429 Rate Limit errors.

1. User checkout latencies spike from 100ms to over 1,300ms, causing browser timeouts.
2. The 10,000 checkout requests/sec completely overwhelm the 500 requests/sec fraud API limit, crashing downstream services and dropping orders.

A Message Queue is a durable, intermediate storage buffer that receives work tasks from Producers, holds them safely in non-volatile storage, and distributes them to Consumer Workers as worker capacity becomes available.

This lesson answers one central question: How do Message Queues execute point-to-point task buffering using Competing Consumers, Visibility Timeouts, and Dead-Letter Queues (DLQs), and how do engineers handle at-least-once delivery duplicate messages using Idempotent Processing?

An Architecture Breakdown: Producer, Broker Queue, and Competing Consumers

A message queue architecture organizes asynchronous task processing around three core components:

flowchart LR
  subgraph Producers
    P1[Checkout API Pod 1]
    P2[Checkout API Pod 2]
    P3[Checkout API Pod 3]
  end
  
  subgraph Durable Broker Buffer
    MQ[(Message Queue Buffer)]
    DLQ[(Dead-Letter Queue - DLQ)]
  end
  
  subgraph Competing Consumers
    W1[Worker 1]
    W2[Worker 2]
    W3[Worker 3]
  end
  
  P1 -->|1. Enqueue Task| MQ
  P2 -->|1. Enqueue Task| MQ
  P3 -->|1. Enqueue Task| MQ
  
  MQ -->|2. Pull Task| W1
  MQ -->|2. Pull Task| W2
  MQ -->|2. Pull Task| W3
  
  W1 -->|3. ACK Success| MQ
  W2 -->|3. NACK / Retries Exceeded| DLQ

Figure 1: Component lifecycle showing Producers enqueuing tasks, Competing Consumers processing work, and Poison Messages routing to a DLQ.

Core Architectural Concepts

1. Producer: The application service that creates task payloads (e.g. SendOrderInvoiceTask) and writes them to the queue broker.
2. Broker Queue: The durable infrastructure layer (e.g. Amazon SQS, RabbitMQ, Redis Streams) that persists messages on disk, manages message visibility, and tracks delivery acknowledgments.
3. Competing Consumers: A pool of worker instances running in parallel, all pulling tasks from the identical queue. Each individual message is delivered to and processed by exactly ONE worker instance in the pool.

Message Processing Lifecycle and Visibility Timeout Mechanics

To guarantee that no message is lost if a consumer worker crashes mid-task, message queues operate using Visibility Timeouts and Acknowledgments (ACKs):

sequenceDiagram
    autonumber
    actor Worker as Consumer Worker 1
    participant Queue as Queue Broker (SQS/RabbitMQ)
    actor Worker2 as Consumer Worker 2
    
    Worker->>Queue: 1. Poll Message (Fetch Task 101)
    Queue->>Queue: 2. Set Visibility Timeout = 30s (Task 101 hidden from other workers)
    Queue-->>Worker: 3. Deliver Task 101 Payload
    
    alt Successful Processing Path
        Worker->>Worker: 4. Process Task 101 (Takes 5 seconds)
        Worker->>Queue: 5. Send ACK (Delete Task 101 from Queue)
    else Worker Crash / Timeout Path
        Worker->>Worker: 4. Worker 1 Crashes / Times Out! (No ACK sent)
        Note over Queue: 5. 30-Second Visibility Timeout Expires!
        Queue->>Queue: 6. Make Task 101 Visible Again
        Worker2->>Queue: 7. Poll Message (Fetch Task 101)
        Queue-->>Worker2: 8. Deliver Task 101 Payload to Worker 2
    end

Figure 2: Sequence diagram demonstrating Visibility Timeout expiration and redelivery upon worker failure.

At-Least-Once Delivery & Idempotency

Due to network partitions and worker retries, modern message queues guarantee At-Least-Once Delivery:

• A single message payload may be delivered to consumer workers more than once.
• For example, if Worker 1 successfully processes Task 101 but experiences a network partition right before sending the ACK signal to the queue, the queue will re-deliver Task 101 to Worker 2 after the visibility timeout expires.

Priority Message Queuing

• Tasks are enqueued with a numeric priority weight (e.g. Priority 1 to 10).
• Broker schedulers ensure that high-priority messages (Priority 10) bypass lower-priority messages in the queue buffer, delivering critical tasks to available consumer workers immediately.

Scaling Consumer Worker Pools (KEDA Auto-Scaling)

• KEDA monitors the queue depth metric (QueueLength) in real time.
• When QueueLength exceeds 1,000 pending messages, KEDA automatically scales the consumer worker pool from 5 pods up to 50 pods.
• Once the backlog drains back down to 0, KEDA scales the worker pool back down to 5 pods, saving infrastructure compute costs.

Queue Deduplication Windows (Amazon SQS FIFO)

• Producers supply a unique MessageDeduplicationId token with every enqueue request.
• If a producer sends a duplicate message payload carrying the same deduplication ID within 5 minutes, the queue broker accepts the write but discards the duplicate payload immediately, guaranteeing zero duplicate tasks in the queue.

Delay Queues and Scheduled Task Execution

• Producers enqueue a message carrying a DeliveryDelay parameter (e.g. 15 minutes).
• The queue broker holds the message in invisible storage until the delay timer expires, preventing workers from polling the task before its scheduled time.

Long Polling vs Short Polling

• Short Polling: Worker queries a subset of broker servers and returns immediately, even if the queue is empty. This generates empty response CPU overhead and unnecessary API billings.
• Long Polling: Worker opens an HTTP/TCP connection that stays open for up to 20 seconds (WaitTimeSeconds = 20), returning as soon as a message arrives. Long polling reduces empty poll responses by up to $98\%$!

The Idempotency Rule

$$\text{Process}(\text{Task}) = \text{Process}(\text{Process}(\text{Task}))$$

flowchart TD
  Receive[Worker Receives Task 101: Order ID 89041] --> CheckDB{Check Idempotency Ledger: Has Task 101 processed?}
  
  CheckDB -->|Yes: Duplicate Message| Skip[Skip Processing & Send ACK Immediately]
  CheckDB -->|No: First Time| Exec[Execute Business Logic & Update DB]
  
  Exec --> Record[Record Task 101 ID in Idempotency Ledger Table]
  Record --> ACK[Send ACK to Queue Broker]

Figure 3: Idempotent Consumer pattern using an Idempotency Ledger table.

Complete Worked Example: Production Go Worker Pool Queue Processor

Let's inspect a complete Go implementation of a Worker Pool Queue Processor for the MessageLab platform (messagelab.com).

package main
import (
	"context"
	"fmt"
	"sync"
	"time"
)
type Task struct {
	ID        string
	Payload   string
	Retries   int
	MaxRetries int
}
type MessageQueueProcessor struct {
	taskQueue   chan Task
	dlqChannel  chan Task
	idempotency map[string]bool
	mu          sync.Mutex
}
func NewMessageQueueProcessor(bufferSize int) *MessageQueueProcessor {
	return &MessageQueueProcessor{
		taskQueue:   make(chan Task, bufferSize),
		dlqChannel:  make(chan Task, bufferSize),
		idempotency: make(map[string]bool),
	}
}
func (p *MessageQueueProcessor) StartWorkerPool(ctx context.Context, numWorkers int) {
	var wg sync.WaitGroup
for i := 1; i <= numWorkers; i++ {
		wg.Add(1)
		go func(workerID int) {
			defer wg.Done()
			p.workerLoop(ctx, workerID)
		}(i)
	}
fmt.Printf("[QUEUE INIT] Started %d Competing Consumer Workers.\n", numWorkers)
}
func (p *MessageQueueProcessor) workerLoop(ctx context.Context, workerID int) {
	for {
		select {
		case <-ctx.Done():
			return
		case task, ok := <-p.taskQueue:
			if !ok {
				return
			}
			p.processTaskWithIdempotency(workerID, task)
		}
	}
}
func (p *MessageQueueProcessor) processTaskWithIdempotency(workerID int, task Task) {
	p.mu.Lock()
	if p.idempotency[task.ID] {
		p.mu.Unlock()
		fmt.Printf("[WORKER %d] Task %s already processed (Idempotent Skip). ACK sent.\n", workerID, task.ID)
		return
	}
	p.mu.Unlock()
// Simulate Processing Logic
	fmt.Printf("[WORKER %d] Processing Task %s (%s)...\n", workerID, task.ID, task.Payload)
	time.Sleep(50 * time.Millisecond)
// Record Idempotency Lock
	p.mu.Lock()
	p.idempotency[task.ID] = true
	p.mu.Unlock()
fmt.Printf("[WORKER %d] Task %s completed successfully. ACK sent.\n", workerID, task.ID)
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Message Queues decouple producers from consumers: Use durable queues to absorb traffic bursts and isolate slow background processing tasks.
2. Competing Consumers process work in parallel: Multiple worker instances pull from the same queue, with each message processed by exactly one worker.
3. Set Visibility Timeouts safely: Ensure visibility timeouts exceed maximum task execution times to prevent premature task redelivery to rival workers.
4. Design consumers for At-Least-Once Idempotency: Assume duplicate message deliveries will occur; use unique task keys to skip duplicate execution.
5. Quarantine broken messages with DLQs: Move poison pill messages that fail $N$ retries to a Dead-Letter Queue to prevent queue pipeline blockages.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• Users upload 4K video files (100 MB to 5 GB).
• Transcoding a video file to 1080p/720p/480p H.264 formats takes 5 to 20 minutes per file.

1. Formulate the queue architecture, worker pool sizing, and Visibility Timeout configuration for the video encoding pipeline.
2. Design the Dead-Letter Queue and retry backoff policy for corrupted video file uploads.

Interactive Self-Assessment

The queue broker assumes the worker crashed and re-delivers the task to a second worker while the first is still processing it.

It automatically formats persistent NVMe SSD disk drives on queue broker servers.

It revokes edge HTTPS TLS encryption certificates on load balancers.

It cuts physical CPU hardware clock speeds in half across all consumer worker nodes.

Consumer workers MUST be Idempotent so that processing duplicate message deliveries produces no unintended side effects.

Consumer workers must convert relational SQL schemas into un-indexed CSV files.

Consumer workers must replace public DNS nameservers with local hosts file entries.

Consumer workers must reboot operating system hypervisors across all worker nodes.

What to Learn Next

• Publish-Subscribe (Pub-Sub) — Fan-Out Event Distribution: Learn 1-to-N event broadcasting and consumer groups.
• Event-Driven Architecture (EDA): Explore microservice decoupling and Transactional Outbox patterns.
• Dead-Letter Queues & Poison Messages: Master poison message quarantine workflows.

Track: Distributed Systems

Series: Kafka & Event Streaming

1. Message Queues — Hand Work Off Without Blocking the User (this guide)
2. Kafka Architecture — Brokers, Partitions, ISR, and Consumers
3. Kafka Partition Ordering and Delivery Guarantees
4. Kafka Exactly-Once Semantics — What Is Actually Guaranteed
5. Transactional Outbox and Saga Patterns — Reliable Multi-Step Work
6. Kafka Streams Introduction
7. Kafka with Spring Boot Code Walkthrough

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab