system-design · intermediate

Publish/Subscribe Messaging — One Event, Many Interested Listeners

The Central Question

Consider an online checkout platform running on the MessageLab platform (messagelab.com) processing 100,000,000 asynchronous events per day:

• When an order is successfully completed, four separate business domains must react to the event:

In a traditional point-to-point architecture, the Checkout API Service must contain explicit code invocation logic for every single downstream service.

When the marketing team launches a new Fraud Machine Learning Service six months later, developers must modify, re-test, and re-deploy the core Checkout API Service source code just to add a fifth downstream call!

Furthermore, if the Analytics Service crashes, its failure risks polluting or blocking the payment checkout flow.

To decouple producers of domain events from consumers of those events, systems deploy Publish/Subscribe Messaging (Pub/Sub).

Publish/Subscribe Messaging is a messaging pattern where Publishers emit named immutable facts called Events to a central Topic, and multiple independent Subscribers register interest to receive their own copy of every published event.

This lesson answers one central question: How does Pub/Sub execute multi-subscriber Fan-Out routing, how do log-based brokers use Partitioning to preserve per-key order, and how do engineering teams manage Schema Evolution without breaking downstream consumers?

Competing Consumers vs. Multi-Subscriber Fan-Out

It is critical to distinguish classic Point-to-Point Message Queues from Publish/Subscribe Topics:

flowchart TD
  subgraph Point-to-Point Message Queue (Single Execution)
    QProducer[Producer] --> Queue[(Task Queue)]
    Queue --> QW1[Worker 1]
    Queue --> QW2[Worker 2]
    Note1["Each task is processed by EXACTLY ONE worker instance."]
  end
  
  subgraph Publish/Subscribe Topic (Fan-Out Copies)
    PProducer[Publisher] --> Topic((Orders Topic))
    Topic --> Sub1[Email Service Subscription]
    Topic --> Sub2[Analytics Service Subscription]
    Topic --> Sub3[Loyalty Service Subscription]
    Note2["EVERY independent subscription receives its OWN COPY of the event!"]
  end

Figure 1: Contrast between point-to-point queue single worker processing and Pub/Sub multi-subscription fan-out.

Architectural Comparison Matrix

Fan-Out Topologies: Topic-to-Queue Routing vs. Partitioned Event Logs

Modern Pub/Sub infrastructure follows two distinct architectural topologies:

1. SNS + SQS Fan-Out Topology (Cloud Message Fan-Out)

flowchart LR
  Pub[Checkout Publisher] --> Topic((SNS Topic: orders.paid))
  
  Topic --> Q1[(SQS Queue: email-service)]
  Topic --> Q2[(SQS Queue: analytics-service)]
  Topic --> Q3[(SQS Queue: loyalty-service)]
  
  Q1 --> W1[Email Workers]
  Q2 --> W2[Analytics Workers]
  Q3 --> W3[Loyalty Workers]

Figure 2: AWS SNS + SQS Fan-Out pattern routing a single topic publish into dedicated service queues.

Partitioning and Message Ordering (Apache Kafka / Event Hubs)

In distributed log-based Pub/Sub brokers (such as Apache Kafka or Azure Event Hubs), topics are partitioned across multiple physical server disks to scale throughput:

flowchart TD
  Pub[Publisher] --> Router{Hash Partition Key: user_id}
  
  Router -->|user_id: 101| P0[(Partition 0 Log)]
  Router -->|user_id: 102| P1[(Partition 1 Log)]
  Router -->|user_id: 103| P2[(Partition 2 Log)]
  
  subgraph Consumer Group: Analytics Service
    C1[Consumer Worker 1] --> P0
    C2[Consumer Worker 2] --> P1
    C3[Consumer Worker 3] --> P2
  end

Figure 3: Partitioned Pub/Sub topic preserving per-key message ordering across a Consumer Group.

The Partition Ordering Guarantee

"Message ordering is strictly guaranteed WITHIN a single partition, but NOT across different partitions."

By setting the partition key to user_id or account_id, all events for that specific user land on the exact same partition in total sequential order (OrderCreated $\rightarrow$ OrderPaid $\rightarrow$ OrderShipped), preserving causal state ordering.

Schema Governance & Schema Registries (Confluent Avro / Protobuf)

• Before publishing an event, the producer registers the event schema (e.g. Apache Avro or Protocol Buffers) with the Schema Registry.
• The Schema Registry enforces strict Backward Compatibility (new schemas can read old events) and Forward Compatibility.
• Event payloads carry a tiny 5-byte schema ID header instead of full JSON keys, reducing network serialization overhead by up to $70\%$!

Zero-Copy Log Streaming Mechanics (Kafka OS Kernel Optimization)

• Instead of copying event data from disk to kernel memory, then to application RAM, then back to kernel socket memory, sendfile streams byte channels directly from the OS page cache to the network interface card (NIC).
• This bypasses CPU context switching, enabling Kafka to saturate physical 10GbE network interfaces with minimal CPU overhead.

Log Compaction for Keyed Event Streams

• Background compaction threads scan log segments and retain only the most recent event record for each unique message key.
• Older update records carrying the same key are purged from disk log files.
• This allows new consumer microservices to restore full current state snapshots by reading the compacted log from offset zero without storing historical noise.

Consumer Group Rebalance Storms

• During a classic eager rebalance, all consumers stop reading, revoke partition assignments, and wait for new partition assignments, creating brief consumption pauses (Rebalance Storms).
• Modern Kafka engines use Cooperative Sticky Assignors, allowing unaffected consumer nodes to continue reading active partitions while only transferring reassigned partitions.

Complete Worked Example: Production Go Topic-Based Pub-Sub Broker

Let's inspect a complete Go implementation of a Topic-Based Pub-Sub Broker for the MessageLab platform (messagelab.com).

package main
import (
	"context"
	"fmt"
	"sync"
)
type Event struct {
	Topic   string
	Key     string
	Payload string
}
type PubSubBroker struct {
	mu          sync.RWMutex
	subscribers map[string][]chan Event
}
func NewPubSubBroker() *PubSubBroker {
	return &PubSubBroker{
		subscribers: make(map[string][]chan Event),
	}
}
func (b *PubSubBroker) Subscribe(topic string, bufferSize int) <-chan Event {
	b.mu.Lock()
	defer b.mu.Unlock()
ch := make(chan Event, bufferSize)
	b.subscribers[topic] = append(b.subscribers[topic], ch)
	fmt.Printf("[SUBSCRIBE] Registered new subscriber for topic '%s'\n", topic)
	return ch
}
func (b *PubSubBroker) Publish(event Event) {
	b.mu.RLock()
	defer b.mu.RUnlock()
subs, exists := b.subscribers[event.Topic]
	if !exists || len(subs) == 0 {
		fmt.Printf("[PUBLISH] No subscribers registered for topic '%s'. Event discarded.\n", event.Topic)
		return
	}
fmt.Printf("[PUBLISH FAN-OUT] Broadcasting event (Key: %s) to %d subscribers on topic '%s'...\n",
		event.Key, len(subs), event.Topic)
for _, sub := range subs {
		select {
		case sub <- event:
		default:
			fmt.Printf("[WARN] Subscriber buffer full on topic '%s'. Dropping event.\n", event.Topic)
		}
	}
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Pub/Sub enables 1-to-Many Fan-Out: Decouple event producers from interested consumers; add new downstream services without altering publisher code.
2. Message ordering is guaranteed per partition: Supply a deterministic partition key (user_id) to direct related events to the same partition.
3. Consumer Groups scale reading in parallel: Divide partitions among consumer group worker instances; max parallel workers equals total partition count.
4. Enforce Backward Schema Compatibility: Use Schema Registries (Avro or Protobuf) to prevent new event fields from crashing downstream consumers.
5. Combine SNS + SQS for isolated service queues: Forward topic events into dedicated SQS queues per service to prevent slow consumers from affecting peers.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• RideCompletedEvent emitted 500 times per second.
• 5 downstream services: Billing, Driver Payouts, Rider Rating, Analytics, and Fraud Detection.

1. Formulate the topic partitioning strategy, partition key selection, and consumer group design.
2. Detail how your schema registry policy prevents breaking changes when the Billing team updates payment payload fields.

Interactive Self-Assessment

A partition is a single append-only disk log file; multiple partitions process in parallel across physical nodes, preserving sequential order only within individual partitions.

Kafka partitioning automatically formats persistent NVMe SSD disk drives across all broker nodes.

Kafka partitioning revokes client HTTPS TLS encryption certificates on load balancers.

Kafka partitioning cuts physical CPU hardware clock speeds in half across all consumer nodes.

Each subscribing service gets a dedicated queue, isolating slow or failing consumers from affecting peer services.

SNS + SQS Fan-Out automatically converts relational database primary key indexes into un-indexed CSV files.

SNS + SQS Fan-Out replaces public DNS nameservers with local hosts file entries.

SNS + SQS Fan-Out reboots operating system hypervisors across all worker nodes.

What to Learn Next

• Event-Driven Architecture (EDA): Master microservice decoupling and Transactional Outbox patterns.
• Change Data Capture (CDC): Learn log-based WAL database event streaming.
• Dead-Letter Queues & Poison Messages: Explore poison message quarantine workflows.

Track: Data, Storage and Messaging

Previous: Event-Driven Architecture — React to Facts, Don’t Chain Calls

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab