system-design · intermediate

Service Discovery — Finding Instances That Change

Start here

Service discovery solves:

“I need to call the payments service—what IPs/ports are healthy right now?”

In modern systems, instances are cattle: containers appear and disappear. Hard-coded addresses break. Discovery provides a name → set of endpoints mapping that updates with membership and health.

You should care because microservices multiply dependencies; without discovery (or an equivalent platform network), deploys and autoscaling become manual pain.

What you will learn

Define service discovery in plain English.
Contrast client-side vs server-side discovery.
See DNS, registries (Eureka/Consul), and platform discovery (K8s).
Connect health checks to removing bad instances.
Work a complete checkout→payments example.
Avoid stale caches and thundering reconnections.

What you should know first

Topic	Why
DNS	Classic name lookup
Load balancing	Often paired with discovery
Client–server	Who looks up whom

Words you need before we begin

Term	Plain English
Service name	Logical name like `payments`.
Instance / endpoint	Concrete address of one process.
Registry	Store of service→instances.
Client-side discovery	Caller queries registry and picks an instance.
Server-side discovery	Caller hits a load balancer that knows instances.
Health check	Probe deciding if instance should receive traffic.
Sidecar / mesh	Helper proxy beside the app for discovery/routing.

Simple story: company directory

Employees change desks. Instead of memorizing seat numbers, you look up “Alice” in the directory (registry) or ask reception (load balancer) to route you. Out-of-date directories send you to empty desks—stale discovery.

The problem without discovery

Config file lists 10.0.0.5:8080 for payments. Autoscaler adds new tasks; old task dies. Callers break until someone edits configs and restarts everything.

Step-by-step explanation

Step 1 — Register on start

Instance announces payments @ 10.0.1.23:8080 with metadata (version, zone).

Step 2 — Heartbeat / lease

Registry removes instances that miss heartbeats (or platform API reflects pod deletion).

Step 3 — Discover

Caller resolves payments to a list or LB VIP.

Step 4 — Select

Round-robin, random, least-conn, or zone-aware choice.

Step 5 — Health gate

Failing readiness checks → removed from pool.

Step 6 — Cache carefully

Clients cache lookups with short TTL; stale caches cause errors after deploys.

Step 7 — Observe

Track discover failures, empty pools, and call error rates after deploys.

Visual mental model

flowchart LR
  SvcA[Service A] -->|resolve payments| Reg[Registry / DNS / platform]
  Reg --> LB[Load balancer or client picker]
  LB --> P1[Payments-1]
  LB --> P2[Payments-2]

Learning question: Who removes an instance that is still registered but returns 500s?

Caption: Health checks and outlier detection—not only registration TTL.

Complete worked example: Kubernetes-style

Starting situation

Checkout pods must call payments pods. Both autoscale.

Decisions

Item	Choice
Mechanism	Kubernetes Service + kube-proxy/IPVS or mesh
Name	`payments.namespace.svc.cluster.local`
Readiness	`/health/ready` checks DB connectivity
Timeouts	Client 200–500ms with retries limited

Flow

Checkout uses cluster DNS to service VIP; platform maps to ready pods. Pod dies → endpoint removal → traffic shifts.

Failures

Readiness always true while app broken → bad deploys (fix probe).
DNS caching in app too long → stale after scale events.
Registry outage in older systems → cached endpoints or fail.

How it works in production

Kubernetes Services/Endpoints
Consul/Eureka in older stacks
Service mesh (Istio/Linkerd) for advanced routing
Cloud LB target groups

Failure modes

Mode	Impact	Mitigation
Stale client cache	Call dead tasks	Short TTL; watch APIs
Missing health checks	Blackhole traffic	Readiness probes
Split registries	Wrong environment calls	Strict config, mTLS
Thundering discover	Registry overload	Caching, platform VIP
Empty pool	Total dependency failure	Alerts; degrade

Trade-offs

Choice	Benefit	Cost
Client-side discovery	Flexible LB logic	Client complexity
Server-side discovery	Simple clients	LB critical path
Platform-native	Less custom code	Platform lock-in
Mesh	Rich policy	Operational weight

Compare with related concepts

Concept	Difference
DNS alone	May lack active health without extras
Load balancing	Uses discovered targets
Service mesh	Continuous discovery + policy
API gateway	North-south edge; discovery is often east-west

Common misunderstandings

“DNS is enough everywhere.” Health and multi-port realities vary.
“Discovery replaces timeouts.” Still need client resilience.
“Register healthy forever.” Must deregister and probe.
“Only microservices need this.” Any dynamic fleet does.
“Client libraries always agree.” Standardize platform patterns.

Check your understanding

What question does discovery answer?
Client-side vs server-side discovery?
Role of readiness probes?
Risk of long client-side DNS TTL?
Name one platform discovery mechanism.

Practice

Draw discovery for three services in a cluster.
Design metadata: version, zone, weight.
Write a runbook for empty endpoint sets.
Compare Consul vs Kubernetes services conceptually.
List metrics for discovery health.

Deeper production notes

Warmup and registration races

Instances may accept traffic before warm. Use readiness gates and gradual traffic.

Multi-cluster

Cross-cluster discovery needs explicit design (gateways, service entries). Do not assume single-cluster DNS works globally.

Additional teaching scenarios

Scenario A — peak load day

Traffic multiplies by ten. Mark which failure modes appear first and the first mitigation for each.

Scenario B — mixed versions

Half the fleet runs an old build. Which assumptions break? Prefer one deploy window of compatibility.

Scenario C — five-sentence teach-back

Explain the core idea without acronyms using only the simple story and worked example.

Scenario D — metrics and alerts

List three metrics and one alert that track user impact or a scarce resource.

Scenario E — non-goals

Name two problems this technique should not solve.

Scenario F — ownership

Who owns dashboards, code, and pages? Blank means not ready for broad rollout.

Revision summary

Discovery maps service names to live instances.
Pair with health checks and careful caching.
Prefer platform-native mechanisms when available.
Client resilience still required.

Glossary

Term	Definition
Service discovery	Dynamic lookup of service endpoints.
Registry	Membership store for instances.
Readiness	Signal that instance can receive traffic.

Abbreviations and terminology

VIP — Virtual IP
DNS — Domain Name System
TTL — Time to live
mTLS — Mutual TLS

What to learn next

Extra teaching notes for first-time builders

Write the single bottleneck you are protecting before picking tools. Name the signal that tells you the design is working for users, not only that internal counters move. When reviewing a change related to this lesson, ask what happens when the component is slow for ten minutes, down entirely, or running twice. Prefer small explicit failure modes that operators can understand at 3 a.m.

Document ownership for dashboards, code, and pages. Undocumented mechanisms become folklore and then outages. Prefer designs that tolerate mixed versions for at least one deploy window so rollouts do not require perfect global simultaneity.

Napkin math helps: estimate peak rate, multiply by payload size, and ask whether the design still holds when a dependency is at half capacity. If the answer depends on luck, add bounds, backpressure, or shedding before production traffic arrives.

Extra teaching notes for first-time builders

FAQ from first-time learners

Q: Is a load balancer the same as discovery?
A: LB distributes among known targets; discovery maintains the target list.

Q: Do serverless functions need discovery?
A: Platforms often invoke by name/ARN—discovery is abstracted.

Q: How fast must updates be?
A: Fast enough for your deploy/scale events; measure error spikes during rollouts.

Track: Software Design and Architecture

Previous: Serverless Architecture — Managed Compute on Demand

Next: Splitting a Monolith Safely (Strangler Fig)

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab