system-design · intermediate

Idempotency for APIs — Safe Retries Without Double Side Effects

The Central Question

Consider a mobile banking application executing a $\$500$ money transfer. The customer taps "Submit Payment".

The mobile app dispatches an HTTP POST /v1/transfers request. The backend server receives the request, debits Account A, credits Account B, and generates a transfer confirmation record.

However, just as the server returns the HTTP 200 OK response payload, a cell tower handoff drops the TCP network connection. The mobile app client experiences a socket timeout. From the client's perspective, the operation failed or timed out. The mobile app automatically retries the request.

If the backend API executes the retry as a new operation, Account A is debited a second time, stealing another $\$500$ from the customer.

Networks are fundamentally unreliable. Packets drop, load balancers time out, and clients retry.

Idempotency is the mathematical property of an API operation where executing the request multiple times with the same intent produces the exact same lasting server state as executing it once.

This lesson answers one central question: How do APIs utilize client-generated Idempotency-Key headers, atomic database locks, and response caching state machines (IN_FLIGHT, COMPLETED, FAILED) to ensure network retries never produce duplicate charges or state mutations?

The Network Drop Hazard: Why Unsafe Retries Cause Data Corruption

In distributed systems, an HTTP request can fail at three distinct points along the execution path:

sequenceDiagram
    autonumber
    actor Client as Client App
    participant GW as API Gateway
    participant API as Payment Service
    participant DB as Database
    
    Client->>GW: 1. POST /v1/transfers ($500)
    GW->>API: 2. Forward Request
    API->>DB: 3. UPDATE accounts SET balance = balance - 500
    DB-->>API: 4. Transaction Committed
    API-->>GW: 5. HTTP 200 OK { transferId: 901 }
    GW--xClient: 6. NETWORK PACKET DROPPED! (Socket Timeout)
    
    Note over Client: Client experiences timeout.<br/>Retries identical POST request.
    
    Client->>GW: 7. RETRY: POST /v1/transfers ($500)
    GW->>API: 8. Forward Request
    API->>DB: 9. UPDATE accounts SET balance = balance - 500
    Note over DB: UN-SAFE RETRY! Second $500 debited!
    DB-->>API: 10. Second Transaction Committed
    API-->>GW: 11. HTTP 200 OK { transferId: 902 }
    GW-->>Client: 12. Client receives 200 OK (User charged $1,000 total)

Figure 1: Sequence diagram illustrating how dropped response packets lead to duplicate charges on un-idempotent endpoints.

The Core Problem

• A failure that occurred before the server executed the mutation (safe to retry).
• A failure that occurred after the server committed the mutation (unsafe to retry).

Method Semantics: Naturally Safe vs. Explicitly Idempotent Operations

HTTP protocols define baseline idempotency expectations for standard methods:

flowchart TD
  Methods[HTTP Protocol Methods] --> Safe[Naturally Safe & Idempotent]
  Methods --> NaturalIdem[Naturally Idempotent but Unsafe]
  Methods --> Unsafe[Unsafe & Non-Idempotent by Default]
  
  Safe --> GET["GET: Read-only query. Safe to retry infinitely."]
  NaturalIdem --> PUT["PUT: Replaces full resource payload. Idempotent."]
  NaturalIdem --> DELETE["DELETE: Removes resource by ID. Idempotent."]
  Unsafe --> POST["POST: Creates resource or side effect. REQUIRES IDEMPOTENCY KEY."]

Figure 2: Taxonomy of HTTP methods categorized by safe vs idempotent protocol rules.

Method Properties Analysis

The Idempotency-Key Pattern and State Machine

To make unsafe POST operations idempotent, clients generate a unique Idempotency Key (typically a UUID v4) for every distinct business intent and transmit it in an HTTP header:

POST /v1/transfers HTTP/1.1
Host: api.checkoutlab.com
Authorization: Bearer eyJhbGci...
Idempotency-Key: 7b9e4a12-890c-4e5f-9a1b-2c3d4e5f6a7b
Content-Type: application/json
{
  "sourceAccountId": "acc_101",
  "destinationAccountId": "acc_202",
  "amount": 50000,
  "currency": "USD"
}

The Server-Side Idempotency State Machine

When a server receives a request with an Idempotency-Key, it evaluates the key against an Atomic Idempotency Store (relational database table or Redis lock):

stateDiagram-v2
    [*] --> NewKey : Key Not Found
    
    NewKey --> IN_FLIGHT : Atomic Insert (Status: IN_FLIGHT)
    note right of IN_FLIGHT
        Lock acquired.
        Execute business logic.
    end note
    
    IN_FLIGHT --> COMPLETED : Work Succeeds
    note right of COMPLETED
        Cache HTTP Status (200)
        + Response JSON payload.
    end note
    
    IN_FLIGHT --> FAILED : Execution Throws Exception
    note right of FAILED
        Store failure status or
        allow key clean-up.
    end note
    
    [*] --> DuplicateKey : Key Already Exists
    
    DuplicateKey --> ReplayResponse : Status == COMPLETED
    note right of ReplayResponse
        Return cached 200 OK + payload
        WITHOUT re-executing logic!
    end note
    
    DuplicateKey --> Conflict409 : Status == IN_FLIGHT
    note right of Conflict409
        Return HTTP 409 Conflict
        (Concurrent attempt in progress)
    end note

Figure 3: State machine governing idempotency key transitions and replay execution.

Database Implementation: Atomic Lock & Response Caching

To guarantee safety across multi-node server pools, checking the key and acquiring the lock must occur in a single atomic database operation.

Relational Database Schema (idempotency_keys)

CREATE TABLE idempotency_keys (
    idempotency_key VARCHAR(255) NOT NULL,
    account_id VARCHAR(64) NOT NULL,
    status VARCHAR(32) NOT NULL, -- 'IN_FLIGHT', 'COMPLETED', 'FAILED'
    response_code INT NULL,
    response_body JSONB NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    locked_until TIMESTAMP WITH TIME ZONE NOT NULL,
    PRIMARY KEY (account_id, idempotency_key)
);

Atomic Execution Blueprint (PostgreSQL Pseudocode)

-- 1. Attempt atomic insert for key
INSERT INTO idempotency_keys (
    idempotency_key, account_id, status, locked_until
) VALUES (
    '7b9e4a12-890c-4e5f-9a1b-2c3d4e5f6a7b', 'acc_101', 'IN_FLIGHT', NOW() + INTERVAL '30 seconds'
)
ON CONFLICT (account_id, idempotency_key) DO NOTHING;
-- 2. Evaluate insert result
-- If Insert Succeeded (1 row inserted):
--    Execute business logic (Debit Account A, Credit Account B).
--    UPDATE idempotency_keys 
--    SET status = 'COMPLETED', response_code = 200, response_body = '{"transferId":"901"}'
--    WHERE idempotency_key = '7b9e4a12-890c-4e5f-9a1b-2c3d4e5f6a7b';
--
-- If Insert Failed (0 rows inserted - Key exists):
--    SELECT status, response_code, response_body FROM idempotency_keys WHERE idempotency_key = '...';
--    If status == 'COMPLETED': Return cached response_code and response_body immediately!
--    If status == 'IN_FLIGHT': Return HTTP 409 Conflict ("Operation currently in progress").

Handling Partial Execution Failures and Outbox Reconciliation

In distributed microservice architectures, an API operation may call third-party payment gateways (such as Stripe or Adyen) before saving state locally.

If the application server process crashes after calling the payment gateway but before updating the local idempotency table to COMPLETED:

flowchart TD
  Crash[Server Crash Mid-Operation] --> Check[Stuck IN_FLIGHT Key Expires: Lease Timeout]
  Check --> Retry[Client Retries Request with Same Idempotency Key]
  Retry --> Recon{Background Reconciliation Job Checks Payment Provider}
  Recon -->|Provider Shows Charge Completed| Complete[Update Local DB to COMPLETED & Return Cached Charge Payload]
  Recon -->|Provider Shows Charge Missing| ReExecute[Re-Try Fresh Charge Attempt Safely]

Figure 4: Reconciliation control loop managing partial execution crashes.

Handling Out-of-Sync Edge Failures

1. Lease Expiration (locked_until): Set a 30-second lock lease on IN_FLIGHT keys. If a node crashes, the lease expires so subsequent retries are not blocked indefinitely.
2. Provider Key Forwarding: Forward the exact same Idempotency-Key to downstream payment gateways so third-party APIs also deduplicate retries.
3. Background Reconciliation Jobs: Run asynchronous workers that query third-party payment APIs for pending idempotency keys before executing new charges.

Execution Sequence: First Request vs. Subsequent Retries

sequenceDiagram
    autonumber
    actor Client as Mobile Client App
    participant API as API Server Node A
    participant APIB as API Server Node B
    participant Store as Idempotency Store (DB)
    participant Core as Core Banking Ledger
    
    Note over Client,Core: Phase 1: First Attempt (Network Drops Response)
    Client->>API: POST /v1/transfers (Header: Idempotency-Key: TX-901)
    API->>Store: INSERT TX-901 (Status: IN_FLIGHT) -> SUCCESS
    API->>Core: Execute Debit $500 (Transfer #8801)
    Core-->>API: 200 OK (Transfer #8801 Committed)
    API->>Store: UPDATE TX-901 (Status: COMPLETED, Payload: { transferId: 8801 })
    API--xClient: HTTP 200 OK (PACKET DROPPED BY NETWORK)
    
    Note over Client,Core: Phase 2: Client Retries identical request to Node B
    Client->>APIB: RETRY: POST /v1/transfers (Header: Idempotency-Key: TX-901)
    APIB->>Store: INSERT TX-901 -> CONFLICT (Row Exists)
    APIB->>Store: SELECT status, response_body WHERE key = TX-901
    Store-->>APIB: Status: COMPLETED, Body: { transferId: 8801 }
    Note over APIB: Bypasses Core Ledger entirely!
    APIB-->>Client: HTTP 200 OK { transferId: 8801 } (Replayed Response)

Figure 5: Sequence diagram contrasting first execution logic against replayed cached responses.

Multi-Tenant Key Scoping and Expiry Retention

Idempotency keys cannot be stored infinitely without exhausting database storage. High-availability idempotency layers govern keys using two critical rules:

flowchart TD
  Key[Incoming Client Idempotency Key] --> Rule1[1. Key Scoping: Account + Key]
  Key --> Rule2[2. Retention TTL Expiry]
  
  Rule1 --> ScopeDesc["Scope keys per Account ID: (account_id, idempotency_key). Prevents cross-tenant key collisions."]
  Rule2 --> TTLDesc["Enforce 24-Hour to 7-Day TTL. Delete completed keys after TTL expires to bound storage growth."]

Figure 6: Operational governing rules for idempotency key scoping and TTL retention.

1. Key Scoping (Tenant Isolation)

2. Retention Time-To-Live (TTL)

If a client retries a request 30 days later with a purged key, the system treats it as a new request. Clients must generate new idempotency keys for new user intents.

Complete Worked Example: CheckoutLab Payment API

Let me detail the production idempotency specification for the CheckoutLab platform (api.checkoutlab.com).

Idempotency Policy Matrix

Failure Modes and Engineering Mitigations

What You Should Remember

1. Idempotency enables safe retries: Networks drop responses. Idempotency guarantees that executing a request multiple times produces the exact same server state as running it once.
2. Clients control key intent: The client generates one Idempotency-Key per logical user action and re-uses that exact key across all retries of that action.
3. Use atomic database locks: Perform key checks and locks in a single atomic database operation (INSERT ... ON CONFLICT) to prevent concurrent race conditions.
4. Cache HTTP status and payload: When a key reaches COMPLETED, store the HTTP status code and JSON response body so replays return identical responses without re-executing logic.
5. Scope keys by tenant account: Always scope keys to (account_id, idempotency_key) to prevent cross-tenant key collision security breaches.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

During a high-concurrency event, two identical requests with Idempotency-Key: TX-8801 reach Server Node A and Server Node B at the exact same millisecond.

Questions:

1. Trace the exact SQL execution sequence on both Server Node A and Server Node B using INSERT ... ON CONFLICT DO NOTHING.
2. Detail how Server Node B handles the request when it discovers the row is currently locked with status = 'IN_FLIGHT'.

Interactive Self-Assessment

It allows the server to identify the retry as the same logical intent and return the cached outcome without double-executing side effects.

It forces the server to create a new database transaction per retry.

It instructs the API Gateway to re-encrypt network TLS sockets.

It converts HTTP POST requests into read-only HTTP GET requests.

User B could send a colliding key string and receive User A's cached private response payload.

The database will crash due to table row index overflow limits.

All API rate limits will be disabled for authenticated users.

The server will fail to return HTTP 409 Conflict status codes.

What to Learn Next

• What Is an API? — Contracts Between Programs: Revisit fundamental HTTP API contracts and status codes.
• API Design — Clear Contracts Clients Can Trust: Explore RESTful resource patterns and RFC 7807 error formatting.
• Rate Limiting Algorithms — Token Bucket, Windows, and Bursts: Master edge traffic governance algorithms.

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab