system-design · intermediate

Leader Election — Picking One Coordinator

Start here

Leader election is how a group of machines agrees on a temporary boss for a role:

“Node X is the leader right now. Everyone else is a follower for this role.”

You need election when exactly one process should coordinate work: running a singleton controller, owning a primary for a shard, writing the head of a replicated log, or deciding failovers.

You should care because without a safe election, two nodes may both act as primary (split brain) and corrupt state. With a poorly tuned election, you may have no leader for too long and make no progress.

What you will learn

1. Define leader election without jargon.
2. Contrast static primary with dynamic election.
3. Use leases, heartbeats, and terms (epochs).
4. Connect election to failover and fencing.
5. Walk a complete multi-pod controller example.
6. Recognize flapping, partitions, and stale leaders.
7. Know when leaderless designs are better.

What you should know first

Words you need before we begin

Simple story: radio on a group hike

Hikers share one radio. They elect who holds it. If the holder falls into a ravine (crashes), the others elect a new radio holder. Two people transmitting opposite directions is split brain—and dangerous.

Where the analogy stops: networks can delay “I still have the radio” messages. Software needs leases and term numbers so a delayed old leader cannot override a new one.

The problem without election

Three replicas all decide independently: “primary database looks dead, I will promote the standby.”

Outcomes:

• All three promote → dual writers.
• None promote → long outage.
• They promote at different times with conflicting ideas of truth.

Step-by-step explanation

Step 1 — Name the exclusive role

Write the invariant: “At most one active reconciler for controller C” or “Exactly one primary for shard 7.”

Step 2 — Choose a coordination backend

Prefer proven primitives: etcd/Kubernetes leases, ZooKeeper, Consul, or consensus inside a database system. Do not invent a cluster membership protocol inside application Redis for money or metadata primaries.

Step 3 — Win a lease

The winner may act as leader only while the lease is valid and renewals succeed.

Step 4 — Renew on a loop

Renew substantially before expiry. If renewals fail, step down and stop exclusive work.

Step 5 — Increase a term on each election

Followers and storage should ignore lower terms (fencing tokens for leadership).

Step 6 — Followers standby

They may serve limited duties (reads) if safe, but must not perform the exclusive write role.

Step 7 — Make work survivable across leadership loss

Idempotent jobs, checkpoints, and compare-and-set updates on resources. Leadership is not a magic shield against double work at the edges of expiry.

Visual mental model

stateDiagram-v2
  [*] --> Follower
  Follower --> Candidate: heartbeat timeout
  Candidate --> Leader: won lease or votes
  Candidate --> Follower: lost election
  Leader --> Follower: lease lost or higher term

Learning question: After a long garbage-collection pause, what must a former leader do before writing again?

Caption: Re-check leadership; never assume the lease survived the pause.

Complete worked example: Kubernetes-style controller

Starting situation

Three pods run a controller that updates custom resources. Dual active controllers may fight and thrash status fields.

Constraints

• Detect dead leader within about fifteen seconds
• Survive rolling deploys
• etcd available for leases

Decisions

Execution

Pod A leads and reconciles. Node failure kills A. Lease expires. Pod B acquires lease and continues from resource state in the API server—not from A’s memory.

Failure samples

1. Brief network blip: renew retries succeed; no election.
2. GC longer than lease: A loses leadership; if A continues, API conflicts or term checks must stop damage.
3. etcd outage: elections cannot complete; alert on leaderless time.

Outcome

Mostly single active controller with automatic failover. Limitation: around lease boundaries you must still design for rare overlap.

How it works in production

• Control planes elect leaders constantly (Kubernetes controllers, operators).
• Data systems elect Raft/Paxos leaders internally—you consume the result as “primary.”
• Application singleton jobs should prefer lease libraries over home-grown heartbeats.

Ownership

Platform owns etcd health; app teams own what they do while leader and how they step down.

Failure modes

Trade-offs

Compare with related concepts

Common misunderstandings

1. “Oldest node should lead.” Unsafe without a protocol.
2. “Leader is permanent.” Leases and failures force re-election.
3. “Leader cannot be wrong.” Leadership ≠ bug-free logic.
4. “Every microservice needs a leader.” Stateless APIs should not.
5. “If ping fails, I am leader.” Uncoordinated local decisions create split brain.

Check your understanding

1. What invariant does leader election protect?
2. Why renew leases instead of holding forever?
3. What is split brain in one sentence?
4. Why do terms/epochs matter?
5. When should you avoid introducing a leader?

Practice

1. Design lease timings for a batch job that runs hourly versus a chat room owner role.
2. List metrics you would graph for a leader-elected worker.
3. Explain fencing when an old leader wakes after a pause.
4. Compare fixed primary with operator-driven failover versus automatic election.
5. Write a runbook step for 'no leader for five minutes'.

Deeper production notes

Capacity and timeouts for leader election

Production systems fail at the edges of timeouts more often than in textbook happy paths. When you deploy leader election, write down the detection interval, the action on failure, and the recovery signal. If any of those three is missing, operators will guess during an incident.

Measure user-visible impact, not only internal counters. A metric that says the mechanism is working while customers cannot complete a journey is a vanity dashboard. Pair technical gauges with at least one journey-level indicator.

Review questions for pull requests

1. What happens if this component is slow for ten minutes?
2. What happens if it is down entirely?
3. What happens if two copies run at once?
4. What happens if clocks are skewed by a few seconds?
5. How will we know in the first five minutes of an incident?

Documentation duties

Link the design to a runbook section: dashboards, common failure signatures, and the first mitigation step (rollback, shed load, step down leader, pause consumers). Undocumented mechanisms become folklore and then outages.

Revision summary

• Election chooses one coordinator for a defined role.
• Leases, heartbeats, and terms keep leadership honest.
• Fence stale leaders; make work idempotent.
• Prefer leaderless designs when exclusivity is unnecessary.
• Watch flapping and leaderless gaps in production.

Glossary

Abbreviations and terminology

• GC — Garbage collection
• HA — High availability
• TTL — Time to live

What to learn next

1. Distributed locks and consensus
2. Failover
3. Consensus algorithms

Additional teaching scenarios

Scenario A — busy day

Imagine traffic multiplies by ten for a marketing event. Re-read the failure modes section and mark which ones become likely first. Write the first mitigation you would take for each marked item. This exercise turns abstract lists into operational instincts.

Scenario B — partial deploy

Half of your instances run the new version and half run the old version. Which assumptions in this lesson break if the two versions disagree about protocols, message fields, or transaction boundaries? Prefer designs that tolerate mixed versions for at least one deploy window.

Scenario C — explain to a new teammate

In five sentences, teach the core idea of this lesson without acronyms. If you cannot, the mental model is not yet solid—revisit the simple story and worked example until the five sentences feel natural.

Scenario D — metric design

List three metrics and one alert threshold you would ship with this mechanism. Good metrics name the user impact or the resource that runs out, not only that a counter incremented.

Scenario E — deliberate non-goals

Write two problems this lesson's technique should not solve. Explicit non-goals prevent cargo-cult adoption where every service gets the same machinery whether it needs it or not.

FAQ from first-time learners

Q: Is the load balancer a leader?
A: No. It distributes requests; it does not grant exclusive write roles for data.

Q: Can a database row implement election?
A: Sometimes via transactional compare-and-set on a lease record—but you must reason carefully about expiry and fencing.

Q: What if a minority partition elects a leader?
A: Correct quorum protocols prevent minorities from winning leadership for the cluster role.

Track: Engineering Foundations

Previous: IP Addresses — How Machines Find Each Other

Next: Load Balancing — Algorithms and Layers

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab