system-design · intermediate

Webhooks — Server-to-Server Event Callbacks

The Central Question

Consider a SaaS payment processing platform running on the EventLab platform (eventlab.com):

• When a customer submits a credit card payment, the payment authorization takes 5 seconds to clear through banking networks.
• A merchant's e-commerce application needs to know the exact millisecond the payment succeeds so it can unlock digital downloads or dispatch shipments.

A Webhook (also known as a Reverse API or HTTP Push Callback) is an event-driven integration mechanism where a Provider Server makes an outbound HTTP POST request to a pre-configured URL endpoint on a Consumer Server immediately when an event occurs.

This lesson answers one central question: How do Webhooks execute real-time server-to-server event delivery, how do engineers authenticate webhooks using HMAC SHA256 cryptographic signatures to prevent tampering and replay attacks, and how do providers manage delivery retries using exponential backoff?

Polling vs. Webhooks Architecture

Webhooks invert the traditional client-server request relationship:

flowchart TD
  subgraph Anti-Pattern: HTTP Polling (Client Pull)
    Client1[Merchant Server] -->|1. GET /status (No Change)| Provider1[(Payment Platform)]
    Client1 -->|2. GET /status (No Change)| Provider1
    Client1 -->|3. GET /status (No Change)| Provider1
    Client1 -->|4. GET /status (SUCCESS!)| Provider1
    Note1["Drawback: 99% Wasted Traffic, High Latency Lag Window"]
  end
subgraph Pattern: Webhook Event Push (Server Push)
    Provider2[(Payment Platform)] -->|Event Occurs: Payment Succeeded| Push["POST https://merchant.com/webhooks/payment<br/>X-Event-Signature: t=17712,v1=a8f9b..."]
    Push --> Listener[Merchant Webhook Listener Endpoint]
    Listener -->|HTTP 200 OK ACK| Provider2
    Note2["Benefit: Sub-Second Real-Time Delivery, Zero Polling Overhead"]
  end

Figure 1: Comparing traditional HTTP polling against real-time Webhook event pushes.

Polling vs Webhook Architectural Matrix

The 4-Stage Webhook Delivery Pipeline

When a domain event occurs in a provider system, the event passes through an automated outbound delivery pipeline:

sequenceDiagram
    autonumber
    actor Customer as User / App
    participant DB as Provider Database
    participant Worker as Outbound Webhook Worker
    participant Consumer as Merchant Webhook Endpoint
    
    Customer->>DB: 1. Complete Payment Transaction
    Note over DB: Payment State -> SUCCESS
    DB->>Worker: 2. Enqueue Outbound Webhook Payload
    
    Worker->>Worker: 3. Compute HMAC SHA256 Signature Header
    Worker->>Consumer: 4. POST https://merchant.com/webhooks (Headers + Body)
    
    alt Successful Delivery Path
        Consumer-->>Worker: 5a. HTTP 200 OK (Processed)
        Worker->>DB: Mark Webhook Delivery SUCCESS
    else Failed / Timeout Path
        Consumer-->>Worker: 5b. HTTP 500 / Timeout Exceeded
        Worker->>Worker: 6. Schedule Exponential Backoff Retry (T + 60s)
    end

Figure 2: Sequence diagram detailing the 4-stage Webhook delivery, signing, and retry pipeline.

Security Protocols: HMAC SHA256 Signatures and Replay Attack Prevention

Because a public Webhook listener URL (https://merchant.com/webhooks) is exposed to the open internet, malicious attackers can send forged POST requests to trick the merchant into shipping goods without payment!

To secure webhooks, providers and consumers use Cryptographic HMAC Signatures and Timestamp Tolerance Limits.

flowchart TD
  subgraph Provider Server (Signing)
    Payload[JSON Event Payload] + Timestamp[Timestamp: 1771239000] --> StringToSign["StringToSign = timestamp + '.' + payload"]
    StringToSign + Secret[Shared Secret Key] --> HMAC[HMAC-SHA256 Algorithm]
    HMAC --> SigHeader["Header: X-Signature: t=1771239000,v1=9f8a7c6..."]
  end
  
  subgraph Consumer Server (Verification)
    RecvHeader[Incoming X-Signature Header] --> CheckTime{Is |t_current - t_header| < 300s?}
    CheckTime -->|NO: Expired| RejectTime[Reject: Replay Attack!]
    CheckTime -->|YES: Valid| RecalcHMAC[Re-compute Local HMAC with Shared Secret]
    RecalcHMAC --> CheckSig{Does Local Signature == v1 Signature?}
    CheckSig -->|NO| RejectSig[Reject: Forged Signature!]
    CheckSig -->|YES| Accept[Accept & Process Payload!]
  end

Figure 3: HMAC SHA256 signature generation and validation verification workflow.

1. HMAC SHA256 Signature Header (X-Signature)

$$\text{Signature} = \text{HMAC-SHA256}(\text{SharedSecret}, \text{Timestamp} + "." + \text{RawBody})$$

The signature is attached in an HTTP header:

POST /webhooks/payment HTTP/1.1
Host: merchant.com
Content-Type: application/json
X-Event-Id: evt_901284
X-Event-Signature: t=1771239000,v1=a8f9c2d1e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1
{
  "event": "payment.succeeded",
  "amountCents": 4999,
  "currency": "USD"
}

2. Preventing Replay Attacks

To stop replay attacks:

1. The signature includes the UNIX timestamp $t$.
2. The consumer verifies that $|T_{\text{current}} - t_{\text{header}}| \le 300\text{ seconds}$ (5-minute tolerance window).
3. If the timestamp is older than 5 minutes, the consumer rejects the request immediately.

Webhook Monitoring and Dead-Letter Quarantine

Webhook Mutual TLS (mTLS) and IP Whitelisting

Webhook Payload Compression & Gzip Negotiation

Retry Policy: Exponential Backoff and Jitter

If a consumer's server experiences a temporary outage or network glitch, the provider must retry delivering the webhook without overwhelming the recovering consumer server.

Providers enforce Exponential Backoff with Jitter:

$$T_{\text{retry}}(c) = (2^c \times T_{\text{base}}) \pm \text{RandomJitter}$$

Where:

• $c$ is the current attempt count ($1, 2, 3, \dots$).
• $T_{\text{base}}$ is the base delay (e.g. $15\text{ seconds}$).

gantt
    title Webhook Exponential Backoff Schedule
    dateFormat ss
    axisFormat %S
    section Initial Attempt
    Attempt 1 (HTTP 503 Outage) :crit, a1, 00, 01
    section Retry Delays
    Delay 1 (15s Backoff) :active, d1, 01, 16
    Attempt 2 (HTTP 503 Outage) :crit, a2, 16, 17
    Delay 2 (30s Backoff) :active, d2, 17, 47
    Attempt 3 (HTTP 200 OK Success!) :done, a3, 47, 48

Figure 4: Timeline diagram illustrating exponential backoff retry spacing.

Complete Worked Example: Go Webhook HMAC Receiver Verification

Let me show you a complete, production-grade Go HTTP webhook handler for the EventLab platform (eventlab.com) that verifies HMAC SHA256 signatures, checks timestamp limits, and enforces idempotency.

package main
import (
	"crypto/hmac"
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"io"
	"net/http"
	"strconv"
	"strings"
	"time"
)
const WebhookSecret = "whsec_771829041_test_secret_key"
func WebhookReceiverHandler(w http.ResponseWriter, r *http.Request) {
	if r.Method != http.MethodPost {
		http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
		return
	}
// 1. Read Raw Body Bytes
	bodyBytes, err := io.ReadAll(r.Body)
	if err != nil {
		http.Error(w, "Bad Request", http.StatusBadRequest)
		return
	}
// 2. Parse X-Event-Signature Header (t=1771239000,v1=hex_signature)
	sigHeader := r.Header.Get("X-Event-Signature")
	if sigHeader == "" {
		http.Error(w, "Missing Signature Header", http.StatusUnauthorized)
		return
	}
timestampStr, signatureHex := parseSignatureHeader(sigHeader)
// 3. Replay Attack Prevention (5-Minute Window)
	timestamp, err := strconv.ParseInt(timestampStr, 10, 64)
	if err != nil || time.Now().Unix()-timestamp > 300 {
		http.Error(w, "Timestamp Expired (Replay Attack)", http.StatusUnauthorized)
		return
	}
// 4. Compute Expected HMAC SHA256
	payloadToSign := fmt.Sprintf("%s.%s", timestampStr, string(bodyBytes))
	mac := hmac.New(sha256.New, []byte(WebhookSecret))
	mac.Write([]byte(payloadToSign))
	expectedSignature := hex.EncodeToString(mac.Sum(nil))
// 5. Constant-Time Signature Comparison
	if !hmac.Equal([]byte(signatureHex), []byte(expectedSignature)) {
		http.Error(w, "Invalid HMAC Signature", http.StatusUnauthorized)
		return
	}
// 6. Idempotent Processing (Return HTTP 200 Fast!)
	eventId := r.Header.Get("X-Event-Id")
	fmt.Printf("[WEBHOOK VERIFIED] Event ID: %s | Processed Successfully!\n", eventId)
w.WriteHeader(http.StatusOK)
	w.Write([]byte({"status":"received"}))
}
func parseSignatureHeader(header string) (string, string) {
	var ts, sig string
	parts := strings.Split(header, ",")
	for _, part := range parts {
		kv := strings.SplitN(part, "=", 2)
		if len(kv) == 2 {
			if kv[0] == "t" {
				ts = kv[1]
			} else if kv[0] == "v1" {
				sig = kv[1]
			}
		}
	}
	return ts, sig
}

Failure Modes and Engineering Mitigations

What You Should Remember

1. Webhooks invert data flow: Webhooks push event notifications in real-time via outbound HTTP POST requests, eliminating inefficient client polling loops.
2. Always verify HMAC SHA256 signatures: Protect webhook listener endpoints by calculating HMAC signatures over timestamp + "." + rawBody using a shared secret.
3. Prevent Replay Attacks with timestamps: Enforce a strict 5-minute timestamp tolerance limit to prevent hackers from replaying intercepted webhook payloads.
4. Acknowledge HTTP 200 fast: Return an HTTP 200 OK response immediately upon receiving a valid webhook; offload heavy processing to an internal task queue.
5. Use Exponential Backoff for retries: Spacing out webhook delivery retries using exponential backoff ($2^c \times T_{\text{base}}$) prevents retry storms during consumer server outages.

Glossary of Terms

Practice Scenario and Self-Assessment

Architecture Scenario

• Merged merchants configure Webhook URLs to receive invoice.paid events.
• Merchant server endpoints experience frequent intermittent outages.

1. Design the retry schedule (initial delay, backoff multiplier, max retry duration) for retrying failed webhook deliveries over a 24-hour window.
2. Detail how your consumer SDK verifies HMAC SHA256 signatures and guards against replay attacks.

Interactive Self-Assessment

It authenticates that the incoming payload originated from the true provider and was not forged or tampered with by an attacker.

It automatically converts XML payloads into JSON format.

It creates B-Tree database indexes on the provider database server.

It forces the client operating system to install missing GPU graphics drivers.

Long processing risks exceeding the provider's HTTP timeout limit (e.g. 5s), causing the provider to mark delivery as failed and trigger duplicate retries.

Returning HTTP 200 immediately closes the primary SQL database connection pool.

Returning HTTP 200 encrypts the payload using AES-256 encryption.

Fast ACKs prevent domain names from expiring at the domain registrar.

What to Learn Next

• Dead-Letter Queues & Poison Messages: Master exponential backoff retry algorithms and poison message quarantine.
• Message Queues — Hand Work Off Without Blocking the User: Learn how to process incoming webhook events asynchronously using internal task queues.
• Pub/Sub — Publish-Subscribe Messaging Architecture: Explore fan-out topic architectures for distributing internal events before delivering external webhooks.

Track: Reliability and Operations

Previous: Thundering Herd

Next: Zero-Downtime Schema Migration

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab