Skip to main content
← PROBLEMS / SYSTEM-DESIGN
An end-to-end domain learning journey

Design a Reliable Multi-Tenant
Webhook Delivery Engine.

A production webhook engine pairs a Transactional Outbox + tenant-partitioned queue with per-tenant concurrency bulkheads, per-endpoint circuit breakers, DNS-pinned SSRF egress guards, HMAC-SHA256 replay-resistant signatures, and tiered retry + DLQ replay.
Start the journey5 stages
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Design a Stripe/GitHub-scale webhook delivery engine dispatching 30,000 outbound events/sec across 200,000 tenant endpoints with HMAC-SHA256 verification, SSRF protection, noisy-neighbor isolation, exponential backoff retries, and Dead Letter Queue (DLQ) replay.

From patterns to decisions.

A production webhook engine pairs a Transactional Outbox + tenant-partitioned queue with per-tenant concurrency bulkheads, per-endpoint circuit breakers, DNS-pinned SSRF egress guards, HMAC-SHA256 replay-resistant signatures, and tiered retry + DLQ replay.

Full walkthrough · 5 stages
01
STAGE 01 · REQUIREMENTS & MATHL5 · APPLIED DESIGN

Clarify Delivery SLOs, Security & Noisy-Neighbor Bounds.

At 30,000 events/sec (~2 KB payload = ~60 MB/s outbound), healthy endpoints respond in <300 ms, while degraded endpoints stall for 5–10s. The engine must guarantee at-least-once delivery with HMAC-SHA256(timestamp + '.' + payload) headers, strict SSRF blocking, and zero cross-tenant head-of-line blocking.

02
STAGE 02 · NAIVE BOTTLENECKSL5 · APPLIED DESIGN

Why Global Worker Pools & Naive URL Fetches Fail.

If 100 global workers pull from one shared queue and Tenant X's webhook hangs for 10s per call, Tenant X quickly occupies all 100 workers—dropping healthy Tenant Y's delivery rate to zero. Separately, validating a hostname with getaddrinfo() and then letting requests.post(url) re-resolve DNS allows DNS rebinding to hit internal 169.254.169.254 cloud metadata IPs.

03
STAGE 03 · ENTERPRISE ARCHITECTUREL6 · SCALE & TRADE-OFFS

Outbox -> Tenant Bulkheads -> Circuit Breakers -> SSRF-Pinned Egress.

Capture events via a Transactional Outbox into a tenant_id-partitioned Kafka topic. Workers enforce per-endpoint concurrency caps (bulkheads) and trip a Circuit Breaker after e.g. 10 consecutive timeouts—routing failing endpoints to delayed retry topics (1m, 5m, 30m, 2h with jitter) and finally a DLQ with manual/API replay. At egress, an SSRF-safe HTTP proxy resolves DNS once, rejects private/link-local CIDRs, and connects directly to the validated IP with the original Host and TLS SNI.

Webhook Dispatch Pipeline

Isolated Multi-Tenant Webhook Delivery & SSRF Guard

Outbox + KafkaKey by tenant_idBulkhead + CBPer-endpoint capHMAC + SSRF GuardIP-pinned HTTPSRetry / DLQJittered backoff

1.Replay-Safe HMAC: Sign t=<unix_ts>.<body> so receivers reject signatures older than 5m.

2.Circuit Breaker: Trips on dead endpoints so workers never stall on 10s timeouts.

3.DNS-Pinned SSRF: Validates resolved IP is public and connects to that IP directly.

Caption: Combining per-endpoint circuit breakers, DNS-pinned SSRF egress guards, and tiered retry topics keeps healthy tenants fast.

import hmac, hashlib, ipaddress, socket, time

def sign_webhook_payload(secret: str, payload: bytes, timestamp: int | None = None) -> tuple[str, int]:
    """Generate replay-resistant Stripe-style HMAC-SHA256 signature header."""
    ts = int(timestamp or time.time())
    signed = f"{ts}.".encode("utf-8") + payload
    sig = hmac.new(secret.encode("utf-8"), signed, hashlib.sha256).hexdigest()
    return f"t={ts},v1={sig}", ts

def resolve_safe_public_ip(hostname: str) -> str:
    """Resolve hostname once and reject private, loopback, or cloud metadata IPs."""
    ip_str = socket.gethostbyname(hostname)
    ip = ipaddress.ip_address(ip_str)
    if ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_multicast or ip.is_reserved:
        raise ValueError(f"Blocked non-public webhook target IP: {ip_str}")
    return ip_str
04
STAGE 04 · RETRIES & DLQ REPLAYL6 · SCALE & TRADE-OFFS

Tiered Delay Queues, Secret Rotation & Customer DLQ Replay.

Never sleep() inside Kafka consumer threads—publish failed deliveries to dedicated delay topics (retry_1m, retry_15m, retry_2h) with jitter, then park exhausted events in a DLQ for 14 days. During secret rotation, emit dual v1 signatures (old + new secret) for 24 hours so customers deploy zero-downtime key updates.

05
STAGE 05 · INTERVIEW FOLLOW-UPSL7 · STAFF / PRINCIPAL

Ordering vs. Throughput & Auto-Disabling Dead Endpoints.

Default to unordered parallel delivery (most webhooks include created_at and event_id for receiver-side idempotency) and auto-disable endpoints that return 100% 4xx/5xx for 7 consecutive days after emailing the tenant.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Question 1 of 3
Score: 0/3

Why must an SSRF guard resolve DNS once, validate the IP, and connect directly to that IP instead of validating the URL and letting the HTTP library re-resolve DNS?

Recommended Learning Flow