Webhooks Explained: Don't Call Us, We'll Call You

Scenario context

Webhooks let one system push event data to another the moment something happens. Instead of polling ("anything new?"), you expose an endpoint and the provider POSTs signed event payloads to you in near real-time. The key production requirements: signature verification, idempotency, async processing.

---

Webhook providers guarantee delivery by retrying on failure — not by ensuring exactly-once. A network timeout af…

This walkthrough focuses on Deep Dive: Why At-Least-Once Delivery Demands Idempotency.

Before making a design choice, clarify:

• Webhook providers guarantee delivery by retrying on failure — not by ensuring exactly-once.
• A network timeout after your handler processes an event but before returning 200 causes the same event to arrive again.
• Idempotency means processing an event twice produces the same result as once.

Concepts in play: api-gateway, event-driven-architecture, stack-and-queue-data-structures, circuit-breaker

Stage 1 — The Basics: HTTP Callbacks and Event-Driven Delivery

• A webhook is an HTTP callback.
• When a specific event occurs, the event source (provider) sends an HTTP POST request to a URL you specify — your webhook endpoint.
• No polling, no long connections.
• The three-step registration flow: 1.

Observable outcome: The three-step registration flow: 1.

Stage 2 — Stop Polling — Let the Provider Ring Your Doorbell

• Polling means your app repeatedly asks a provider "anything new?" — burning bandwidth and adding latency whether or not anything changed.
• Webhooks invert the call: you register a URL, and the provider calls your endpoint the instant an event occurs.
• Analogy: Polling is calling the courier every 5 minutes.
• A webhook is the courier ringing your doorbell the moment the package arrives.

Observable outcome: A webhook is the courier ringing your doorbell the moment the package arrives.

Stage 3 — What a Webhook Payload Looks Like

• Most providers send a structured JSON body over HTTPS POST: Key fields in every real-world webhook: - id — unique event identifier; use for deduplication.
• • type — event name; drives routing in your handler.
• • created — Unix timestamp; enables replay-window validation.
• • data — the event payload; schema varies by event type.

{
  "id":      "evt_101",
  "type":    "payment.succeeded",
  "created": 1772877602,
  "data": {
    "transaction_id": "txn_9001",
    "amount":         4999,
    "currency":       "USD"
  }
}

Observable outcome: - data — the event payload; schema varies by event type.

Stage 4 — Webhook Delivery Flow

• The complete lifecycle from event to processed business logic: The critical design principle: your endpoint's only job is to acknowledge receipt (return 200) as fast as possible.
• All actual work happens asynchronously.
• This prevents provider retry storms.
• Provider retry schedules (common defaults): Providers consider a delivery successful only when they receive a 2xx response within their timeout window (usually 5–30 seconds).

flowchart TD
    A[Event occurs at Provider] --> B[Provider signs payload with HMAC]
    B --> C[Provider POSTs to your endpoint over HTTPS]
    C --> D{Signature valid?}
    D -->|No| E[Return 401  reject]
    D -->|Yes| F{Event ID already seen?}
    F -->|Yes| G[Return 200  silently ignore]
    F -->|No| H[Persist raw payload to DB]
    H --> I[Enqueue job for async worker]
    I --> J[Return 200 immediately]
    J --> K[Worker executes business logic]
    K --> L[Mark event as processed]

Observable outcome: Provider retry schedules (common defaults): Providers consider a delivery successful only when they receive a 2xx response within their timeout window (usually 5–30 seconds).

Stage 5 — The Production-Safe Webhook Handler

• A naive endpoint that just processes inline is dangerous: it creates duplicate actions when providers retry.
• The correct pattern has 5 ordered steps: Why return 200 before processing?
• Most providers retry if they don't receive a timely 2xx.
• If your business logic runs inline and takes too long, the same event fires twice.

flowchart TD
    A[Provider POST] --> B{Valid HMAC
signature?}
    B -->|No| C[Return 401]
    B -->|Yes| D{Duplicate
event_id?}
    D -->|Yes| E[Return 200  ignore]
    D -->|No| F[Persist event
to durable store]
    F --> G[Enqueue for async
worker]
    G --> H[Return 200 immediately]
    H --> I[Worker processes
business logic]

// Node.js / Express — production-safe handler skeleton
app.post('/webhooks/provider', express.raw({ type: 'application/json' }), (req, res) => {
  const sig = req.header('X-Signature');
  if (!isValidHmac(req.body, sig, process.env.WEBHOOK_SECRET)) {
    return res.status(401).send('invalid signature');
  }

  const event = JSON.parse(req.body.toString('utf8'));

  if (isDuplicate(event.id)) {
    return res.status(200).send('duplicate ignored');
  }

  persistEvent(event);    // write to DB before ack
  enqueueEvent(event);    // hand off to async worker
  return res.status(200).send('accepted');
});

Observable outcome: If your business logic runs inline and takes too long, the same event fires twice.

Stage 6 — Real-World Applications: Where Webhooks Power Real Systems

Observable outcome: You can explain how “Real-World Applications: Where Webhooks Power Real Systems” changes the architecture or implementation path.

End-to-end outcome

Webhook providers guarantee delivery by retrying on failure — not by ensuring exactly-once.

Alternatives considered

Decision Guide: When Webhooks Are (and Aren't) the Answer

Decision Guide: When Webhooks Are (and Aren't) the Answer

Failure and degraded modes

Trade-offs & Failure Modes: Failure Modes You Must Defend Against

• Detection: Most webhook providers use at-least-once delivery — duplicates are normal, ordering is not guaranteed.
• User or operator impact: Most webhook providers use at-least-once delivery — duplicates are normal, ordering is not guaranteed.