Your checkout platform has a 4-hop synchronous call graph: `API Gateway -> Order Service -> Inventory Service -> Pricing Service -> Postgres`. Every service uses default client retries (`maxAttempts=3`, fixed 100 ms delay, no jitter, no circuit breaker). At 2,000 RPS, a brief 15% Postgres brownout causes Pricing timeouts. Within 45 seconds, Pricing receives 162,000 RPS of retry traffic, thread pools saturate across all 4 tiers, and the system remains 100% down even 20 minutes after Postgres recovers. Stop the metastable collapse and design a retry-safe architecture.
From patterns to decisions.
Your checkout platform has a 4-hop synchronous call graph: `API Gateway -> Order Service -> Inventory Service -> Pricing Service -> Postgres`. Every service uses default client retries (`maxAttempts=3`, fixed 100 ms delay, no jitter, no circuit breaker). At 2,000 RPS, a brief 15% Postgres brownout causes Pricing timeouts. Within 45 seconds, Pricing receives 162,000 RPS of retry traffic, thread pools saturate across all 4 tiers, and the system remains 100% down even 20 minutes after Postgres recovers. Stop the metastable collapse and design a retry-safe architecture.
4 tiers of 3x retries.
4 tiers of 3x retries create amplification; the resulting backlog and synchronized 100 ms retries keep CPU/queues saturated in a self-sustaining metastable loop.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Monitoring `upstream_rq_retry / upstream_rq_total` catches amplification before total collapse. |
Step 1
Measure upstream_rq_retry vs upstream_rq_total in Envoy across all 4 tiers.
Step 2
Identify that retry traffic (162k RPS) exceeds Postgres recovery capacity.
Step 3
Confirm that removing load (shedding) is required to exit the metastable state.
Alternative 1
Check x-envoy-attempt-count headers in access logs.
Failure Mode 1
Restarting Pricing pods one by one fails because the 162k RPS retry wave immediately kills each new pod.
curl -s localhost:9901/stats/prometheus | grep -E 'upstream_rq_retry|upstream_rq_pending_overflow'
Centralize retries at the edge.
Centralize retries at the edge with a 20% retry_budget, full-jitter backoff, and explicit deadline checks.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Edge-only retry + 20% budget trades a tiny amount of wasted upstream hops for guaranteed immunity to $R^N$ storms. |
Set `budget_percent
20.0 and min_retry_concurrency: 5 in Envoy circuit_breakers`.
Step 2
Disable retries in internal hops (num_retries: 0).
Step 3
Add deadline check in Java before attempting any retry.
Alternative 1
Use Envoy Retry-On: 5xx only with retry_priority and PreviousHosts predicate.
Failure Mode 1
Retrying on 503 when 503 is used for load-shedding creates a feedback loop.
retry_budget:
budget_percent:
value: 20.0
min_retry_concurrency: 5
A circuit breaker trips binary.
A circuit breaker trips binary (all-or-nothing) at a threshold, whereas a 20% retry budget smoothly throttles retries proportionally to healthy traffic.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Smooth degradation prevents sudden cliff-edge oscillations from circuit breakers flapping open/closed. |
Step 1
At 5% failure, 20% budget allows all 5% to retry.
Step 2
At 50% failure, 20% budget caps retries at 20% and fast-fails the rest.
Step 3
Pair with a circuit breaker as the final backstop at >60% failure.
Alternative 1
Use adaptive concurrency limiting (Netflix Concurrency Limits / Vegas) at each service.
Failure Mode 1
A badly tuned circuit breaker with minimumNumberOfCalls=10 can flap rapidly under bursty traffic.
def allow_retry(active_reqs: int, active_retries: int, budget_pct: float = 0.20) -> bool:
return active_retries < max(5, int(active_reqs * budget_pct))
Implementation & Guardrails (4).
Require a client-generated Idempotency-Key stored in Postgres/Redis with an atomic INSERT ... ON CONFLICT or state-machine lock.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Adds one indexed DB/Redis lookup (~2 ms) in exchange for safe retries on payment mutations. |
Step 1
Reject POST /orders without Idempotency-Key header (400 Bad Request).
Step 2
Store (idempotency_key, request_hash, response_json, status) in Postgres.
Step 3
Return cached response on replay; return 409 Conflict if request_hash differs.
Alternative 1
Disable retries completely on POST and let the mobile client prompt the user.
Failure Mode 1
Retrying a POST when the first attempt actually committed on the DB but timed out on the response socket.
INSERT INTO idempotency_keys (key, request_hash, status)
VALUES ($1, $2, 'IN_PROGRESS')
ON CONFLICT (key) DO NOTHING
RETURNING key;
Inject a 30-second 50% fault/latency.
Inject a 30-second 50% fault/latency spike in staging using Chaos Mesh or Envoy fault injection and verify recovery within 15 seconds of fault removal.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Game-day fault injection proves recovery behavior before production incidents occur. |
Apply 50% `delay
1500ms` on Pricing Service for 30s at 2,000 RPS.
Step 2
Verify Pricing RPS never exceeds 2,400 RPS.
Step 3
Remove fault and confirm p99 returns to < 60 ms within 15 seconds.
Alternative 1
Use Toxiproxy between Pricing and Postgres in a staging load test.
Failure Mode 1
Testing at 10% load instead of 100% peak load misses queue-buildup metastable loops.
http_filters:
- name: envoy.filters.http.fault
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault
delay: { percentage: { numerator: 50, denominator: HUNDRED }, fixed_delay: 1.5s }
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Replace fixed retries with exponential backoff + full jitter, enforce a 20% token-bucket retry budget, and retry only at the outermost edge.
- 02Trade-off tip: Smooth degradation prevents sudden cliff-edge oscillations from circuit breakers flapping open/closed.
- 03Watch out for: Retrying a
POSTwhen the first attempt actually committed on the DB but timed out on the response socket.
Preventing Retry Storms Jitter Budgets — Normal → Load / Fault → Recovery
Replace fixed retries with exponential backoff + full jitter, enforce a 20% token-bucket retry budget, and retry only at the outermost edge.
Putting Preventing Retry Storms with Jitter & Retry Budgets Together
Replace fixed retries with exponential backoff + full jitter, enforce a 20% token-bucket retry budget, and retry only at the outermost edge.
Putting Preventing Retry Storms with Jitter & Retry Budgets Together
- Set
budget_percent: 20.0andmin_retry_concurrency: 5in Envoycircuit_breakers. - Disable retries in internal hops (
num_retries: 0). - Add deadline check in Java before attempting any retry.
Interview Readiness Check
Require a client-generated Idempotency-Key stored in Postgres/Redis with an atomic INSERT ... ON CONFLICT or state-machine lock.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Preventing Retry Storms with Jitter & Retry Budgets
Replace fixed retries with exponential backoff + full jitter, enforce a 20% token-bucket retry budget, and retry only at the outermost edge.
