kafka · advanced

Kafka Exactly-Once Semantics — What Is Actually Guaranteed

Start here

Marketing says exactly-once. Engineers must ask: exactly once what?

In Kafka, exactly-once semantics (EOS) usually means:

• A produce retry does not create duplicate records in the log (idempotent producer / transactions).
• A consume → process → produce pipeline can commit offsets and output records atomically with Kafka transactions—so the pipeline does not double-emit on crash in the happy configuration.

What you will learn

1. Separate at-least-once from EOS claims.
2. Explain idempotent producers.
3. Sketch transactional read-process-write.
4. Know isolation levels for consumers reading transactional data.
5. Design external side effects safely.
6. Avoid compliance overclaims.

Words you need before we begin

Simple story: factory conveyor with a stamp

If the stamp machine retries after a power blip, an idempotent stamp does not double-mark the same widget on the belt. If the machine also ships a postcard for each widget, the postcard system still needs its own “do not mail twice” rule—that is outside the belt’s exactly-once.

The problem without clarity

Teams enable a flag named exactly-once, call Stripe in the consumer, and still double-charge because the external call is not part of the Kafka transaction.

Step-by-step (conceptual)

Step 1 — Start from at-least-once

Default mental model for consumers: duplicates possible.

Step 2 — Idempotent producer

Helps when producers retry after network uncertainty so the log does not gain duplicates for those retries (within broker/producer capabilities).

Step 3 — Transactions for multi-partition atomic writes

Producer begins transaction, sends to outputs, sends offset commits to internal topics, commits transaction.

Step 4 — Consumers use read_committed when needed

So they do not see aborted transaction data.

Step 5 — External systems

Use idempotency keys, unique constraints, or outbox/inbox patterns. Treat EOS as intra-Kafka unless proven otherwise.

Step 6 — Performance and ops costs

Transactions add overhead and operational constraints (fencing zombie producers). Measure.

Step 7 — Test crash scenarios

Kill workers between side effect and commit; verify outcomes.

Visual mental model

flowchart LR
  In[Input topic] --> App[Transactional app]
  App --> Out[Output topic]
  App --> Off[Offset commit in txn]
  App -.->|not atomic with Kafka| Ext[External API / DB]

Learning question: Which arrow is not covered by Kafka EOS alone?

Caption: External side effects need their own safety story.

Complete worked example

A fraud scorer reads payments, writes payments-scored. With transactions, a crash does not leave output without offset advance or offset advance without output (for the Kafka parts). When it also updates a Redis dashboard counter, that counter still needs idempotent increments keyed by event id.

Failure modes

Trade-offs

Common misunderstandings

1. “Exactly-once everywhere.” No—define the boundary.
2. “Idempotent producer = EOS business process.” Insufficient alone.
3. “Transactions replace idempotency keys to Stripe.” They do not.
4. “Turn on EOS, ignore testing.” Crash tests required.
5. “All consumers must use transactions.” Only when requirements justify.

Check your understanding

1. What does idempotent producer protect?
2. What is read-process-write?
3. Why external API calls break naive EOS stories?
4. What is read_committed for?
5. Name one ops cost of transactions.

Practice

1. Draw a pipeline with and without external DB.
2. Write an idempotent email send keyed by eventId.
3. List crash points and outcomes.
4. Compare outbox vs Kafka transactions for DB+event.
5. Read current Kafka docs section on EOS and note version.

Deeper production notes

Version sensitivity

EOS behavior and configuration names have evolved—pin Kafka client/broker versions and re-verify.

Specialist boundary

For regulated processing guarantees, involve specialists; do not rely on blog-level claims.

Additional teaching scenarios

Scenario A — 10× peak

Scenario B — dependency outage

Scenario C — teach-back

Revision summary

• EOS in Kafka is real but bounded.
• Great for Kafka-to-Kafka pipelines with transactions.
• External effects need idempotency/reconciliation.
• Prefer honest language over slogans.

Glossary

Abbreviations and terminology

• EOS — Exactly-once semantics
• PID — Producer id (internal to idempotence)
• TLS — For secure clusters (orthogonal but required in prod)

What to learn next

1. Partition ordering and delivery
2. Kafka architecture
3. Outbox and sagas
4. Payment idempotency

Extra teaching notes

When you apply this lesson, write the user-visible success metric first, then the failure mode you fear most. Design the smallest mechanism that protects that metric under partial failure. Prefer explicit timeouts, idempotency, and ownership over adding more infrastructure boxes.

In interviews or design reviews, narrate assumptions, request paths, and trade-offs out loud. A correct-enough design with clear failure handling beats a buzzword diagram without numbers. Revisit the worked example and restate it for a different domain to prove you own the ideas, not the template wording.

Extra teaching notes

When you apply this lesson, write the user-visible success metric first, then the failure mode you fear most. Design the smallest mechanism that protects that metric under partial failure. Prefer explicit timeouts, idempotency, and ownership over adding more infrastructure boxes.

In interviews or design reviews, narrate assumptions, request paths, and trade-offs out loud. A correct-enough design with clear failure handling beats a buzzword diagram without numbers. Revisit the worked example and restate it for a different domain to prove you own the ideas, not the template wording.

Extra teaching notes

When you apply this lesson, write the user-visible success metric first, then the failure mode you fear most. Design the smallest mechanism that protects that metric under partial failure. Prefer explicit timeouts, idempotency, and ownership over adding more infrastructure boxes.

In interviews or design reviews, narrate assumptions, request paths, and trade-offs out loud. A correct-enough design with clear failure handling beats a buzzword diagram without numbers. Revisit the worked example and restate it for a different domain to prove you own the ideas, not the template wording.

Extra teaching notes

When you apply this lesson, write the user-visible success metric first, then the failure mode you fear most. Design the smallest mechanism that protects that metric under partial failure. Prefer explicit timeouts, idempotency, and ownership over adding more infrastructure boxes.

In interviews or design reviews, narrate assumptions, request paths, and trade-offs out loud. A correct-enough design with clear failure handling beats a buzzword diagram without numbers. Revisit the worked example and restate it for a different domain to prove you own the ideas, not the template wording.

Extra teaching notes

When you apply this lesson, write the user-visible success metric first, then the failure mode you fear most. Design the smallest mechanism that protects that metric under partial failure. Prefer explicit timeouts, idempotency, and ownership over adding more infrastructure boxes.

In interviews or design reviews, narrate assumptions, request paths, and trade-offs out loud. A correct-enough design with clear failure handling beats a buzzword diagram without numbers. Revisit the worked example and restate it for a different domain to prove you own the ideas, not the template wording.

Interview and production field guide for this topic

Use this section as deliberate practice, not filler. Rewrite the worked example for a second domain you know well—fintech, education, logistics, or media. Keep the same skeleton: requirements, estimates, high-level diagram, request path, data model, scale lever, failure modes, and trade-offs. If you cannot fill every section without copying buzzwords, you do not yet own the design.

Numbers to force yourself to state

Always speak order-of-magnitude figures: peak QPS, storage growth per day, fan-out factor, connection counts, or queue depth. Wrong numbers that are explicit beat silent hand-waving. Correct the numbers when the interviewer or teammate challenges them; that is collaboration, not failure.

Failure minute

Set a timer for sixty seconds and list only failures: timeouts, duplicates, hot keys, dependency outages, bad deploys, and data corruption paths. For each, name detection and first mitigation. Designs that only describe the happy path are incomplete for production and weak in interviews.

Ownership and operability

Name the dashboard, the alert, the runbook section, and the team that pages. If any are blank, the system will train you during an incident. Prefer progressive delivery: canaries, flags, and rollback notes written before the change lands.

Consistency and retries

State whether the design assumes at-least-once delivery, whether handlers are idempotent, and where unique constraints live. Retries without idempotency are how double charges, double messages, and duplicate fan-out jobs appear. Timeouts without bounds are how thread pools die.

What good looks like in a review

A strong design review or interview answer clarifies scope, makes assumptions audible, draws a minimal path, deepens one or two bottlenecks, and closes with trade-offs and evolution. Use that bar on kafka-exactly-once-semantics round 0 every time you revisit it.

FAQ from first-time learners

Q: Should every service enable transactions?
A: No—use when duplicate pipeline emissions are costly and scope is Kafka-internal.

Q: Does EOS remove consumer group duplicates forever?
A: It addresses specific crash/retry classes—still design carefully.

Track: Data, Storage and Messaging

Previous: Kafka Architecture — Brokers, Partitions, ISR, and Consumers

Next: Kafka Partition Ordering and Delivery Guarantees

Series: Kafka & Event Streaming

1. Message Queues — Hand Work Off Without Blocking the User
2. Kafka Architecture — Brokers, Partitions, ISR, and Consumers
3. Kafka Partition Ordering and Delivery Guarantees
4. Kafka Exactly-Once Semantics — What Is Actually Guaranteed (this guide)
5. Transactional Outbox and Saga Patterns — Reliable Multi-Step Work
6. Kafka Streams Introduction
7. Kafka with Spring Boot Code Walkthrough

All series

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab