Design a Stripe/GitHub-scale webhook delivery engine dispatching 30,000 outbound events/sec across 200,000 tenant endpoints with HMAC-SHA256 verification, SSRF protection, noisy-neighbor isolation, exponential backoff retries, and Dead Letter Queue (DLQ) replay.
From patterns to decisions.
A production webhook engine pairs a Transactional Outbox + tenant-partitioned queue with per-tenant concurrency bulkheads, per-endpoint circuit breakers, DNS-pinned SSRF egress guards, HMAC-SHA256 replay-resistant signatures, and tiered retry + DLQ replay.
Clarify Delivery SLOs, Security & Noisy-Neighbor Bounds.
At 30,000 events/sec (~2 KB payload = ~60 MB/s outbound), healthy endpoints respond in <300 ms, while degraded endpoints stall for 5–10s. The engine must guarantee at-least-once delivery with HMAC-SHA256(timestamp + '.' + payload) headers, strict SSRF blocking, and zero cross-tenant head-of-line blocking.
Why Global Worker Pools & Naive URL Fetches Fail.
If 100 global workers pull from one shared queue and Tenant X's webhook hangs for 10s per call, Tenant X quickly occupies all 100 workers—dropping healthy Tenant Y's delivery rate to zero. Separately, validating a hostname with getaddrinfo() and then letting requests.post(url) re-resolve DNS allows DNS rebinding to hit internal 169.254.169.254 cloud metadata IPs.
Outbox -> Tenant Bulkheads -> Circuit Breakers -> SSRF-Pinned Egress.
Capture events via a Transactional Outbox into a tenant_id-partitioned Kafka topic. Workers enforce per-endpoint concurrency caps (bulkheads) and trip a Circuit Breaker after e.g. 10 consecutive timeouts—routing failing endpoints to delayed retry topics (1m, 5m, 30m, 2h with jitter) and finally a DLQ with manual/API replay. At egress, an SSRF-safe HTTP proxy resolves DNS once, rejects private/link-local CIDRs, and connects directly to the validated IP with the original Host and TLS SNI.
Isolated Multi-Tenant Webhook Delivery & SSRF Guard
1.Replay-Safe HMAC: Sign t=<unix_ts>.<body> so receivers reject signatures older than 5m.
2.Circuit Breaker: Trips on dead endpoints so workers never stall on 10s timeouts.
3.DNS-Pinned SSRF: Validates resolved IP is public and connects to that IP directly.
Caption: Combining per-endpoint circuit breakers, DNS-pinned SSRF egress guards, and tiered retry topics keeps healthy tenants fast.
import hmac, hashlib, ipaddress, socket, time
def sign_webhook_payload(secret: str, payload: bytes, timestamp: int | None = None) -> tuple[str, int]:
"""Generate replay-resistant Stripe-style HMAC-SHA256 signature header."""
ts = int(timestamp or time.time())
signed = f"{ts}.".encode("utf-8") + payload
sig = hmac.new(secret.encode("utf-8"), signed, hashlib.sha256).hexdigest()
return f"t={ts},v1={sig}", ts
def resolve_safe_public_ip(hostname: str) -> str:
"""Resolve hostname once and reject private, loopback, or cloud metadata IPs."""
ip_str = socket.gethostbyname(hostname)
ip = ipaddress.ip_address(ip_str)
if ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_multicast or ip.is_reserved:
raise ValueError(f"Blocked non-public webhook target IP: {ip_str}")
return ip_str
Tiered Delay Queues, Secret Rotation & Customer DLQ Replay.
Never sleep() inside Kafka consumer threads—publish failed deliveries to dedicated delay topics (retry_1m, retry_15m, retry_2h) with jitter, then park exhausted events in a DLQ for 14 days. During secret rotation, emit dual v1 signatures (old + new secret) for 24 hours so customers deploy zero-downtime key updates.
Ordering vs. Throughput & Auto-Disabling Dead Endpoints.
Default to unordered parallel delivery (most webhooks include created_at and event_id for receiver-side idempotency) and auto-disable endpoints that return 100% 4xx/5xx for 7 consecutive days after emailing the tenant.
Practice: Make the trade-off yourself.
Why must an SSRF guard resolve DNS once, validate the IP, and connect directly to that IP instead of validating the URL and letting the HTTP library re-resolve DNS?