Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Preventing Retry Storms with
Jitter & Retry Budgets.

Your checkout platform has a 4-hop synchronous call graph: API Gateway -> Order Service -> Inventory Service -> Pricing Service -> Postgres. Every service uses default client retries (maxAttempts=3, fixed 100 ms delay, no jitter, no circuit breaker). At 2,000 RPS, a brief 15% Postgres brownout causes Pricing timeouts. Within 45 seconds, Pricing receives 162,000 RPS of retry traffic, thread pools saturate across all 4 tiers, and the system remains 100% down even 20 minutes after Postgres recovers. Stop the metastable collapse and design a retry-safe architecture.
Start the journey5 stages
Loading diagram…
Preventing Retry Storms with Jitter & Retry Budgets architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your checkout platform has a 4-hop synchronous call graph: `API Gateway -> Order Service -> Inventory Service -> Pricing Service -> Postgres`. Every service uses default client retries (`maxAttempts=3`, fixed 100 ms delay, no jitter, no circuit breaker). At 2,000 RPS, a brief 15% Postgres brownout causes Pricing timeouts. Within 45 seconds, Pricing receives 162,000 RPS of retry traffic, thread pools saturate across all 4 tiers, and the system remains 100% down even 20 minutes after Postgres recovers. Stop the metastable collapse and design a retry-safe architecture.

From patterns to decisions.

Your checkout platform has a 4-hop synchronous call graph: `API Gateway -> Order Service -> Inventory Service -> Pricing Service -> Postgres`. Every service uses default client retries (`maxAttempts=3`, fixed 100 ms delay, no jitter, no circuit breaker). At 2,000 RPS, a brief 15% Postgres brownout causes Pricing timeouts. Within 45 seconds, Pricing receives 162,000 RPS of retry traffic, thread pools saturate across all 4 tiers, and the system remains 100% down even 20 minutes after Postgres recovers. Stop the metastable collapse and design a retry-safe architecture.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

4 tiers of 3x retries.

4 tiers of 3x retries create 34=81×3^4 = 81\times amplification; the resulting backlog and synchronized 100 ms retries keep CPU/queues saturated in a self-sustaining metastable loop.

ChoiceWhat it buysWhat it costs
Trade-off 1Monitoring `upstream_rq_retry / upstream_rq_total` catches amplification before total collapse.

Step 1

Measure upstream_rq_retry vs upstream_rq_total in Envoy across all 4 tiers.

Step 2

Identify that retry traffic (162k RPS) exceeds Postgres recovery capacity.

Step 3

Confirm that removing load (shedding) is required to exit the metastable state.

Alternative 1

Check x-envoy-attempt-count headers in access logs.

Failure Mode 1

Restarting Pricing pods one by one fails because the 162k RPS retry wave immediately kills each new pod.

curl -s localhost:9901/stats/prometheus | grep -E 'upstream_rq_retry|upstream_rq_pending_overflow'
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Centralize retries at the edge.

Centralize retries at the edge with a 20% retry_budget, full-jitter backoff, and explicit deadline checks.

ChoiceWhat it buysWhat it costs
Trade-off 1Edge-only retry + 20% budget trades a tiny amount of wasted upstream hops for guaranteed immunity to $R^N$ storms.

Set `budget_percent

20.0 and min_retry_concurrency: 5 in Envoy circuit_breakers`.

Step 2

Disable retries in internal hops (num_retries: 0).

Step 3

Add deadline check in Java before attempting any retry.

Alternative 1

Use Envoy Retry-On: 5xx only with retry_priority and PreviousHosts predicate.

Failure Mode 1

Retrying on 503 when 503 is used for load-shedding creates a feedback loop.

retry_budget:
  budget_percent:
    value: 20.0
  min_retry_concurrency: 5
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

A circuit breaker trips binary.

A circuit breaker trips binary (all-or-nothing) at a threshold, whereas a 20% retry budget smoothly throttles retries proportionally to healthy traffic.

ChoiceWhat it buysWhat it costs
Trade-off 1Smooth degradation prevents sudden cliff-edge oscillations from circuit breakers flapping open/closed.

Step 1

At 5% failure, 20% budget allows all 5% to retry.

Step 2

At 50% failure, 20% budget caps retries at 20% and fast-fails the rest.

Step 3

Pair with a circuit breaker as the final backstop at >60% failure.

Alternative 1

Use adaptive concurrency limiting (Netflix Concurrency Limits / Vegas) at each service.

Failure Mode 1

A badly tuned circuit breaker with minimumNumberOfCalls=10 can flap rapidly under bursty traffic.

def allow_retry(active_reqs: int, active_retries: int, budget_pct: float = 0.20) -> bool:
    return active_retries < max(5, int(active_reqs * budget_pct))
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (4).

Require a client-generated Idempotency-Key stored in Postgres/Redis with an atomic INSERT ... ON CONFLICT or state-machine lock.

ChoiceWhat it buysWhat it costs
Trade-off 1Adds one indexed DB/Redis lookup (~2 ms) in exchange for safe retries on payment mutations.

Step 1

Reject POST /orders without Idempotency-Key header (400 Bad Request).

Step 2

Store (idempotency_key, request_hash, response_json, status) in Postgres.

Step 3

Return cached response on replay; return 409 Conflict if request_hash differs.

Alternative 1

Disable retries completely on POST and let the mobile client prompt the user.

Failure Mode 1

Retrying a POST when the first attempt actually committed on the DB but timed out on the response socket.

INSERT INTO idempotency_keys (key, request_hash, status)
VALUES ($1, $2, 'IN_PROGRESS')
ON CONFLICT (key) DO NOTHING
RETURNING key;
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Inject a 30-second 50% fault/latency.

Inject a 30-second 50% fault/latency spike in staging using Chaos Mesh or Envoy fault injection and verify recovery within 15 seconds of fault removal.

ChoiceWhat it buysWhat it costs
Trade-off 1Game-day fault injection proves recovery behavior before production incidents occur.

Apply 50% `delay

1500ms` on Pricing Service for 30s at 2,000 RPS.

Step 2

Verify Pricing RPS never exceeds 2,400 RPS.

Step 3

Remove fault and confirm p99 returns to < 60 ms within 15 seconds.

Alternative 1

Use Toxiproxy between Pricing and Postgres in a staging load test.

Failure Mode 1

Testing at 10% load instead of 100% peak load misses queue-buildup metastable loops.

http_filters:
  - name: envoy.filters.http.fault
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault
      delay: { percentage: { numerator: 50, denominator: HUNDRED }, fixed_delay: 1.5s }

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Replace fixed retries with exponential backoff + full jitter, enforce a 20% token-bucket retry budget, and retry only at the outermost edge.
  • 02Trade-off tip: Smooth degradation prevents sudden cliff-edge oscillations from circuit breakers flapping open/closed.
  • 03Watch out for: Retrying a POST when the first attempt actually committed on the DB but timed out on the response socket.
Behavioral Progression · Preventing Retry Storms Jitter Budgets

Preventing Retry Storms Jitter Budgets — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Replace fixed retries with exponential backoff + full jitter, enforce a 20% token-bucket retry budget, and retry only at the outermost edge.

Synthesis / One connected design

Putting Preventing Retry Storms with Jitter & Retry Budgets Together

Replace fixed retries with exponential backoff + full jitter, enforce a 20% token-bucket retry budget, and retry only at the outermost edge.

A candidate architecture, not a universal answer

Putting Preventing Retry Storms with Jitter & Retry Budgets Together

Step 1
Step 2
Step 3
  1. Set budget_percent: 20.0 and min_retry_concurrency: 5 in Envoy circuit_breakers.
  2. Disable retries in internal hops (num_retries: 0).
  3. Add deadline check in Java before attempting any retry.
Require a client-generated `Idempotency-Key` stored in Postgres/Redis with an atomic `INSERT ... ON CONFLICT` or state-machine lock.
Interview Readiness Check

Require a client-generated Idempotency-Key stored in Postgres/Redis with an atomic INSERT ... ON CONFLICT or state-machine lock.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Preventing Retry Storms with Jitter & Retry Budgets

Replace fixed retries with exponential backoff + full jitter, enforce a 20% token-bucket retry budget, and retry only at the outermost edge.