payments · advanced
Payment Idempotency and Reconciliation
Start here
In payments, retries are guaranteed. Mobile networks drop after a successful charge. Users double-tap. Webhooks redeliver. Workers crash after side effects.
Idempotency means the same logical payment attempt does not create multiple captures.
Reconciliation means periodically comparing your ledger to provider reports and fixing gaps (missing webhooks, unknown states).
Together they are how serious systems avoid silent money loss and double charges.
What you will learn
- Model payment intents vs charges.
- Apply idempotency keys end to end.
- Handle unknown outcomes (timeouts).
- Design webhook handlers safely.
- Build reconciliation jobs.
- Know limits and specialist boundaries.
Words you need before we begin
| Term | Plain English |
|---|---|
| Payment intent | Your record of a user’s attempt to pay. |
| Idempotency key | Client/server token for one logical attempt. |
| Provider reference | Provider’s id for a charge/payment. |
| Capture / authorize | Take funds vs hold funds (scheme-dependent). |
| Webhook | Provider HTTP callback about status changes.
| Ledger | Internal money movement records. |
|---|---|
| Reconciliation | Matching internal vs external records. |
| Unknown state | You timed out; provider may have succeeded. |
Simple story: duplicate coffee orders
You tap pay; the app spins; you tap again. Without idempotency, two charges. With idempotency, the second tap returns the first charge. At day end, the shop compares the terminal report to the till—that is reconciliation.
The problem
Double charge
Timeout after provider success + retry without key → two captures.
Lost confirmation
Provider succeeded; your DB write failed; user sees failure; support chaos.
Webhook races
Webhook arrives before your API response handling finishes; order of updates must be safe.
Step-by-step design
Step 1 — Create PaymentIntent first
Durable row: intent_id, amount, currency, customer, status=CREATED, idempotency_key unique.
Step 2 — Call provider with deterministic key
Pass idempotency key / key derived from intent id so provider dedupes.
Step 3 — Persist provider reference and status transitions
State machine: CREATED → AUTHORIZED/CAPTURED/FAILED/CANCELLED. Only allow valid transitions.
Step 4 — Timeouts become investigate
On timeout: leave PENDING; do not invent success; background reconcile with provider GET by reference or key.
Step 5 — Webhooks are at-least-once
Verify signatures; idempotently apply status; ack only after durable update.
Step 6 — Ledger entries
Append-only ledger lines linked to intent; never edit history—compensate with reversing entries.
Step 7 — Reconciliation jobs
Daily (or hourly): download provider settlements; match to ledger; open cases for unmatched; alert on gaps beyond threshold.
Visual mental model
sequenceDiagram
participant C as Client
participant API as Payments API
participant L as Ledger DB
participant P as Provider
C->>API: pay + Idempotency-Key
API->>L: insert intent
API->>P: charge with key
P-->>API: charge_id OR timeout
API->>L: update status
P-->>API: webhook
API->>L: idempotent status apply
Note over L,P: reconcilers compare reports
Learning question: What should the client do on timeout—new key or same key?
Caption: Same key until terminal status is known.
Complete worked example
Checkout for $25.99, key idem_7f. First attempt times out after provider captured. Client retries with idem_7f. Your API finds intent CAPTURED with provider ref ch_123 and returns success without a second capture. Next morning reconcilers confirm ch_123 appears once in provider export and once in ledger.
Failure modes
| Mode | Risk | Mitigation |
|---|---|---|
| New key per retry | Double capture | Client SDK rules |
| Trusting only webhooks | Missed events | Active provider fetch |
| Editing balances in place | Audit nightmare | Append-only ledger |
| Ignoring partial refunds | Drift | Explicit refund objects |
| Manual ops without tools | Human error | Case UI + dual control |
Trade-offs
| Choice | Benefit | Cost |
|---|---|---|
| Authorize then capture | Control | Complexity |
| Auto-capture | Simple | Harder cancel |
| Strong sync ledger | Clarity | Latency |
| Async settlement awareness | Scale | Temporary uncertainty UX |
Common misunderstandings
- “Provider idempotency alone is enough.” You still need local intent state.
- “Webhook = source of truth only.” Combine with APIs and reports.
- “Exactly-once messaging fixes payments.” Money needs ledger discipline.
- “Failed in UI means no charge.” Unknown ≠ failed.
- “Reconciliation is accounting-only.” It is a core reliability control.
Check your understanding
- Why create an intent before calling the provider?
- What is an unknown payment state?
- How should webhooks be secured and applied?
- What does reconciliation compare?
- Why append-only ledger?
Practice
- Sequence for double-tap pay with one key.
- Design tables: intents, attempts, ledger_entries.
- Write reconciling pseudocode for unmatched provider charges.
- Draft user-visible copy for pending payment.
- List metrics: duplicate key hits, pending age, recon gaps.
Deeper production notes
Multi-rail realities
Cards, wallets, bank transfers differ in finality timing. SEPA-like rails need longer reconciliation windows—see SEPA lesson.
Specialist boundary
Scheme rules, chargebacks, and regulatory reporting need specialist review. This lesson is engineering foundation, not legal advice.
Additional teaching scenarios
Scenario A — 10× peak
Which component saturates first? First mitigation?Scenario B — dependency down 30 minutes
What still works? What degrades?Scenario C — five-sentence wrap
Requirements, core mechanism, scale lever, failure mode, trade-off.Revision summary
- Idempotency keys + intents + state machines stop double charges.
- Unknown timeouts require fetch/reconcile, not guesswork.
- Webhooks are untrusted at-least-once events.
- Reconciliation closes the loop with provider truth.
- Money systems need ledgers and human case workflows.
Glossary
| Term | Definition |
|---|---|
| Payment intent | Internal record of a pay attempt. |
| Reconciliation | Matching internal ledger to provider reports. |
| Unknown state | Outcome not yet known after uncertainty. |
Abbreviations and terminology
- SDK — Software development kit
- HMAC — Hash-based message authentication (webhook signatures)
- UTC — Time basis for reports
What to learn next
Extra teaching notes
When you apply this lesson, write the user-visible success metric first, then the failure mode you fear most. Design the smallest mechanism that protects that metric under partial failure. Prefer explicit timeouts, idempotency, and ownership over adding more infrastructure boxes.
In interviews or design reviews, narrate assumptions, request paths, and trade-offs out loud. A correct-enough design with clear failure handling beats a buzzword diagram without numbers. Revisit the worked example and restate it for a different domain to prove you own the ideas, not the template wording.
Extra teaching notes
When you apply this lesson, write the user-visible success metric first, then the failure mode you fear most. Design the smallest mechanism that protects that metric under partial failure. Prefer explicit timeouts, idempotency, and ownership over adding more infrastructure boxes.
In interviews or design reviews, narrate assumptions, request paths, and trade-offs out loud. A correct-enough design with clear failure handling beats a buzzword diagram without numbers. Revisit the worked example and restate it for a different domain to prove you own the ideas, not the template wording.
Extra teaching notes
When you apply this lesson, write the user-visible success metric first, then the failure mode you fear most. Design the smallest mechanism that protects that metric under partial failure. Prefer explicit timeouts, idempotency, and ownership over adding more infrastructure boxes.
In interviews or design reviews, narrate assumptions, request paths, and trade-offs out loud. A correct-enough design with clear failure handling beats a buzzword diagram without numbers. Revisit the worked example and restate it for a different domain to prove you own the ideas, not the template wording.
Extra teaching notes
When you apply this lesson, write the user-visible success metric first, then the failure mode you fear most. Design the smallest mechanism that protects that metric under partial failure. Prefer explicit timeouts, idempotency, and ownership over adding more infrastructure boxes.
In interviews or design reviews, narrate assumptions, request paths, and trade-offs out loud. A correct-enough design with clear failure handling beats a buzzword diagram without numbers. Revisit the worked example and restate it for a different domain to prove you own the ideas, not the template wording.
Extra teaching notes
When you apply this lesson, write the user-visible success metric first, then the failure mode you fear most. Design the smallest mechanism that protects that metric under partial failure. Prefer explicit timeouts, idempotency, and ownership over adding more infrastructure boxes.
In interviews or design reviews, narrate assumptions, request paths, and trade-offs out loud. A correct-enough design with clear failure handling beats a buzzword diagram without numbers. Revisit the worked example and restate it for a different domain to prove you own the ideas, not the template wording.
Interview and production field guide for this topic
Use this section as deliberate practice, not filler. Rewrite the worked example for a second domain you know well—fintech, education, logistics, or media. Keep the same skeleton: requirements, estimates, high-level diagram, request path, data model, scale lever, failure modes, and trade-offs. If you cannot fill every section without copying buzzwords, you do not yet own the design.
Numbers to force yourself to state
Always speak order-of-magnitude figures: peak QPS, storage growth per day, fan-out factor, connection counts, or queue depth. Wrong numbers that are explicit beat silent hand-waving. Correct the numbers when the interviewer or teammate challenges them; that is collaboration, not failure.
Failure minute
Set a timer for sixty seconds and list only failures: timeouts, duplicates, hot keys, dependency outages, bad deploys, and data corruption paths. For each, name detection and first mitigation. Designs that only describe the happy path are incomplete for production and weak in interviews.
Ownership and operability
Name the dashboard, the alert, the runbook section, and the team that pages. If any are blank, the system will train you during an incident. Prefer progressive delivery: canaries, flags, and rollback notes written before the change lands.
Consistency and retries
State whether the design assumes at-least-once delivery, whether handlers are idempotent, and where unique constraints live. Retries without idempotency are how double charges, double messages, and duplicate fan-out jobs appear. Timeouts without bounds are how thread pools die.
What good looks like in a review
A strong design review or interview answer clarifies scope, makes assumptions audible, draws a minimal path, deepens one or two bottlenecks, and closes with trade-offs and evolution. Use that bar on payment-idempotency-reconciliation round 0 every time you revisit it.
FAQ from first-time learners
Q: Can we use only database unique constraints?
A: Necessary but not sufficient—you still need provider keys and reconcilers.
Q: Who owns reconciliation breaks?
A: Payments/ops with engineering tooling—define explicitly.
Track: Domain Architecture
Next: SEPA Lifecycle & Reconciliation
Series: Idempotency & Exactly-Once
- Idempotency, Retries & Backoff
- Duplicate Requests & the Idempotency Gap
- Exactly-Once Processing vs Practical Deduplication
- Payment Idempotency and Reconciliation (this guide)
By Shubham Jain