distributed-systems · intermediate

Distributed Locks and Consensus — Coordinating Without a Single Boss Memory

Start here

On one machine, a mutex (mutual exclusion lock) stops two threads from editing the same structure at once.

Across many machines, there is no shared memory. If two API instances both try to “become the job runner” or “update the same wallet,” you need a distributed coordination mechanism.

Two related ideas:

1. Distributed lock — a way to try to ensure only one client holds a named lock at a time (with important caveats).
2. Consensus — a protocol for replicas to agree on a value/order (leader election, replicated logs) even when some nodes fail or networks partition.

What you will learn

1. Why single-machine locks do not work across processes.
2. Use leases/TTLs and fencing tokens.
3. See when locks are the wrong tool (prefer DB constraints).
4. Get an intuition for consensus (agree despite failures).
5. Work a complete “only one billing cron” example.
6. Know failure modes: lock expiry mid-work, clock issues, unsafe Redis recipes.

What you should know first

Words you need before we begin

Simple story: bathroom key at a hostel

One physical key means one bathroom user. If someone disappears with the key, a time limit (lease) and front-desk reissue matter. If two forged keys exist, you get awkward collisions—like split brain.

Where the analogy stops: networks can delay “I still have the key” messages; software needs fencing so an old key cannot open the vault after a new key was issued.

The problem without coordination

Two app instances run a cron: “charge all due subscriptions.”

Without coordination:

• Both charge customer C → double charge.

• Instance A holds lock, freezes GC for 60s.
• Lock TTL expires; B acquires lock and charges.
• A wakes and also charges → still double charge.

Step-by-step: safer distributed locking mindset

Step 1 — Prefer database uniqueness when it fits

Unique constraint on (subscriptionId, period) for invoices often beats a lock.

Step 2 — If you need a lock, use a system designed for it

ZooKeeper/etcd/Consul-style coordination, or carefully reviewed libraries. Treat home-grown Redis locks as dangerous unless you deeply understand failure modes.

Step 3 — Always use a lease (TTL)

Dead processes must not hold forever.

Step 4 — Issue a fencing token on acquire

Storage operations for the critical resource should reject lower tokens.

Example: A acquires token 5, pauses; B acquires token 6; A’s write with token 5 is rejected.

Step 5 — Keep critical sections short

Do not hold a lock while calling slow third parties if you can avoid it.

Step 6 — Make work idempotent anyway

Locks fail. Idempotency is belt-and-suspenders.

Step 7 — Do not use locks for large-scale throughput control

Locks serialize; they can become bottlenecks. Design less shared mutable state.

Step-by-step: consensus intuition (beginner)

Step 1 — Goal

Nodes agree on “what is the next command?” or “who is leader?” so replicas converge.

Step 2 — Majority quorums

If you require a majority to accept a value, two different values cannot both get majorities in the same term (simplified Raft intuition).

Step 3 — Leaders

Many systems elect a leader to order writes, then replicate the log.

Step 4 — What consensus does not do

It does not make your business logic correct, remove the need for timeouts, or magically span arbitrary multi-company transactions.

Visual mental model

sequenceDiagram
  participant A as Instance A
  participant L as Lock service
  participant S as Storage
  participant B as Instance B
A->>L: acquire job-lock
  L-->>A: OK token=5
  Note over A: GC pause / long work
  L->>L: lease expires
  B->>L: acquire job-lock
  L-->>B: OK token=6
  B->>S: write with token=6
  S-->>B: accept
  A->>S: write with token=5
  S-->>A: reject stale token

Learning question: What goes wrong if storage ignores fencing tokens?

Caption: Lock service alone is not enough; the resource must enforce tokens.

Complete worked example: single active scheduler

Starting situation

Three API replicas. One should run “nightly invoice job,” not three.

Constraints

• Double invoice forbidden
• Job may run up to 10 minutes
• Process can freeze or die mid-job

Decisions

Execution

Leader runs job. If leader dies, new leader elected. Unique constraints prevent double invoices even if both overlap briefly.

Failure behavior

• Clock skew on clients: do not use local clocks for correctness of TTL ownership—use the coordination service’s lease mechanism.
• Long GC without renew: leadership lost; design for that.

Outcome

Mostly one active scheduler + hard data constraints. Limitation: still need monitoring for “no leader” and stuck jobs.

How it works in production

• Kubernetes leader election for controllers
• etcd/ZooKeeper for metadata coordination
• Database advisory locks for coarser app-level tasks (know session semantics)
• Consensus inside Kafka/controller, Cockroach, etc.—you use it indirectly

Ownership

Platform often runs etcd; app teams own how they use locks and must document TTL and fencing.

Failure modes

Trade-offs

Compare with related concepts

Common misunderstandings

1. “I locked in Redis, so I am safe.”

1. “TTL long enough means no double holders.”

1. “Consensus means the cluster never wrong.”

1. “Locks fix distributed transactions.”

1. “We need consensus for every app feature.”

Check your understanding

1. Why can two processes both think they hold a lock after a pause?
2. What is a fencing token for?
3. When are unique constraints better than locks?
4. What does consensus help replicas do?
5. Why renew leases during long jobs?

Practice

1. Design fencing for a file-writer critical section.
2. Rewrite double-billing prevention with only DB constraints.
3. List three metrics for lock systems (hold time, acquire failures, expiries).
4. Explain split brain to a junior using two primary databases.
5. Critique a blog post that says SET key nx ex 30 is enough for safety.

Revision summary

• Distributed locks coordinate critical sections across machines.
• Use leases, fencing tokens, and idempotent work.
• Prefer data constraints when they solve the real bug.
• Consensus helps nodes agree (leaders, logs) under failures.
• Easy APIs hide hard failure modes—design defensively.

Glossary

Abbreviations and terminology

• TTL — Time To Live
• CAS — Compare And Set
• GC — Garbage Collection
• mutex — Mutual exclusion lock

What to learn next

1. Leader election
2. Distributed locking and lease expiry
3. Consistency models
4. Quorum reads vs writes

Deeper production notes

Redlock and internet arguments (practical takeaway)

There is long-standing debate about Redis-based locking algorithms under pauses and clock issues. Practical guidance for this curriculum: do not invent a lock in Redis from a blog snippet for money or uniqueness. Prefer database constraints, or coordination systems with well-understood leases, or reviewed libraries—and still design idempotent work + fencing where writes matter.

Observability for locks

Track acquire latency, acquire failures, hold time histograms, forced expiries, and “work continued after loss of leadership.” The last metric catches the GC-pause double-worker class of bugs.

Consensus as a dependency

When you depend on etcd/ZK, that cluster’s quorum loss blocks leader election and can freeze controllers. Treat coordination systems as critical infrastructure with backups, monitoring, and change control—not a sidecar afterthought.

FAQ from first-time learners

Q: Should I implement Raft myself?
A: Almost never. Use battle-tested coordination services and databases.

Q: Is ZooKeeper required?
A: No. Many teams use etcd, DB locks, or cloud primitives—pick with failure-mode literacy.

Q: Can I lock across payment provider calls?
A: Holding distributed locks across slow external I/O is risky; prefer idempotent provider APIs and short local critical sections.

Track: Distributed Systems

Previous: Quorum Reads vs Quorum Writes

Next: Transactional Outbox and Saga Patterns — Reliable Multi-Step Work

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab