Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Deadline Propagation &
Request Timeout Budgets.

Your checkout platform routes traffic through Envoy Gateway (30s edge timeout) -> OrderService (45s HTTP client timeout) -> InventoryService (60s gRPC timeout, 3 retries with 5s backoff) -> PricingDB (30s query timeout). During a 12% slow-down on PricingDB, Envoy returns 504 Gateway Timeout at 30s, triggering mobile client retries, while OrderService and InventoryService keep threads and DB connections pinned for up to 195s. CPU hits 98% and connection pools exhaust within 90 seconds. Redesign the timeout, deadline propagation, and retry architecture.
Start the journey5 stages
Loading diagram…
Deadline Propagation & Request Timeout Budgets architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your checkout platform routes traffic through Envoy Gateway (30s edge timeout) -> OrderService (45s HTTP client timeout) -> InventoryService (60s gRPC timeout, 3 retries with 5s backoff) -> PricingDB (30s query timeout). During a 12% slow-down on PricingDB, Envoy returns 504 Gateway Timeout at 30s, triggering mobile client retries, while OrderService and InventoryService keep threads and DB connections pinned for up to 195s. CPU hits 98% and connection pools exhaust within 90 seconds. Redesign the timeout, deadline propagation, and retry architecture.

From patterns to decisions.

Your checkout platform routes traffic through Envoy Gateway (30s edge timeout) -> OrderService (45s HTTP client timeout) -> InventoryService (60s gRPC timeout, 3 retries with 5s backoff) -> PricingDB (30s query timeout). During a 12% slow-down on PricingDB, Envoy returns 504 Gateway Timeout at 30s, triggering mobile client retries, while OrderService and InventoryService keep threads and DB connections pinned for up to 195s. CPU hits 98% and connection pools exhaust within 90 seconds. Redesign the timeout, deadline propagation, and retry architecture.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Envoy closes the client connection.

Envoy closes the client connection at 30s, but OrderService (45s) and InventoryService (60s x 3 retries) have no knowledge of the edge deadline and keep executing zombie work.

ChoiceWhat it buysWhat it costs
factorPer-hop static timeouts vs propagated budgets; pro: Static timeouts are simple to configure in isolation.; con: They inevitably drift out of sync and create timeout inversion.

Step 1

Map each hop's configured timeout and retry count from edge to database.

Step 2

Calculate worst-case downstream execution time: 3 retries * 60s + backoffs = 195s.

Step 3

Observe that client retries on 504 multiply concurrent pinned threads by 3-5x.

Alternative 1

Fail-fast circuit breakers (complementary, but do not stop in-flight slow requests before the breaker trips).

name

Connection Pool Starvation; description: HikariCP pools fill with abandoned queries, causing even healthy fast endpoints to fail.; mitigation: Propagate deadlines down to JDBC and abort queries when the caller disconnects.

# Broken configuration causing timeout inversion
edge_envoy_timeout: 30s
order_service_http_timeout: 45s       # > edge!
inventory_grpc_timeout: 60s         # > order!
inventory_retries: 3                # up to 180s+ zombie work
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Implementation & Guardrails (2).

Use nanoTime-based relative deadline tracking in Java and Envoy's native retry_budget + x-envoy-upstream-rq-timeout-ms.

ChoiceWhat it buysWhat it costs
factor50ms Hop Safety Buffer; pro: Ensures downstream errors serialize cleanly back to the caller before the caller's timer fires.; con: Reduces usable leaf budget by 50ms per hop (150ms across 3 hops).

Step 1

Parse incoming deadline duration into a monotonic System.nanoTime() deadline.

Step 2

Subtract 50ms hop overhead before making downstream gRPC/HTTP calls.

Step 3

Bind remaining ms to MySQL MAX_EXECUTION_TIME and skip retries if budget < 2 * p99.

Alternative 1

OpenTelemetry Baggage for custom HTTP frameworks where gRPC metadata is unavailable.

name

Missing Header Fallback; description: Internal batch callers without headers could get infinite timeouts.; mitigation: Default missing headers to 5,000ms at service ingress.

public final class DeadlineContext {
  private final long deadlineNanos;

  public DeadlineContext(long budgetMs) {
    this.deadlineNanos = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(budgetMs);
  }

  public long remainingMs() {
    return Math.max(0L, TimeUnit.NANOSECONDS.toMillis(deadlineNanos - System.nanoTime()));
  }

  public <T extends AbstractStub<T>> T attachToGrpcStub(T stub, long hopBufferMs) {
    long budget = remainingMs() - hopBufferMs;
    if (budget <= 0) {
      throw Status.DEADLINE_EXCEEDED.withDescription("Budget exhausted").asRuntimeException();
    }
    return stub.withDeadlineAfter(budget, TimeUnit.MILLISECONDS);
  }
}
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Relative durations eliminate NTP clock.

Relative durations eliminate NTP clock skew vulnerabilities across distributed nodes while only costing a small, predictable hop-overhead subtraction.

ChoiceWhat it buysWhat it costs
factorRelative Duration vs Epoch Timestamp; pro: Zero dependency on synchronized wall clocks.; con: Does not account for variable network queuing delay unless a small buffer is subtracted.

Step 1

Compare 150ms NTP skew against a 300ms remaining budget: absolute timestamps cause false rejections.

Step 2

Show how relative durations only depend on local monotonic clocks (System.nanoTime()).

Alternative 1

Synchronized TrueTime / AWS Time Sync Service (microsecond accuracy, but unavailable across hybrid/client networks).

name

Unbounded Retry Storms; description: Without retry_budget, per-request retries still double traffic during brownouts.; mitigation: Enforce 20% global retry budget in Envoy.

# Envoy Route Policy with Retry Budget
route:
  cluster: order_service
  timeout: 3.0s
  retry_policy:
    retry_on: 5xx,reset,connect-failure
    num_retries: 1
    per_try_timeout: 1.2s
    retry_budget:
      budget_percent: 20.0
      min_retry_concurrency: 10
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (4).

Application-layer thread interruption does not automatically kill an in-flight MySQL query; you must bind the deadline to the SQL driver and DB engine.

ChoiceWhat it buysWhat it costs
factorDriver Statement.cancel() vs MAX_EXECUTION_TIME; pro: MAX_EXECUTION_TIME runs inside the MySQL server even if the client network drops.; con: MAX_EXECUTION_TIME applies only to read-only SELECTs, not DML.

Step 1

Set JDBC Statement.setQueryTimeout() from the remaining deadline.

Step 2

Inject MySQL /+ MAX_EXECUTION_TIME(ms) / hint for sub-second precision.

Step 3

Listen to gRPC Context.current().addListener() to call Statement.cancel() immediately on client disconnect.

Alternative 1

ProxySQL query timeout rules at the database proxy tier.

name

Orphaned DML Locks; description: Long UPDATE/DELETE statements ignore MAX_EXECUTION_TIME.; mitigation: Use Statement.cancel() via gRPC cancellation listener + innodb_lock_wait_timeout=5.

Context.current().addListener(ctx -> {
  try { preparedStatement.cancel(); } catch (SQLException ignored) {}
}, cancelExecutor);
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Inject synthetic latency faults in.

Inject synthetic latency faults in staging/canary using Envoy fault injection and assert that leaf SQL queries abort within 100ms of the edge deadline.

ChoiceWhat it buysWhat it costs
factorCanary Chaos Injection vs Synthetic Load Tests; pro: Tests real production Envoy + HikariCP + MySQL behavior.; con: Must be scoped strictly to dark-launch or synthetic test headers (x-chaos-test: true).

Step 1

Enable Envoy HTTP fault injection (5s fixed delay on 1% of canary PricingDB calls).

Verify Prometheus metrics

grpc_server_handled_total{grpc_code='DeadlineExceeded'} and zero HikariCP pool saturation.

Step 3

Alert on retry_budget_exhausted_total spikes.

Alternative 1

Automated CI integration test using Toxiproxy to inject 5s TCP latency.

name

Leaking Chaos Faults to Real Users; description: Unscoped fault percentage impacts live checkout conversion.; mitigation: Gate fault injection on a dedicated header match (x-synthetic-probe: true).

# Envoy Fault Injection for Chaos Testing
typed_per_filter_config:
  envoy.filters.http.fault:
    delay:
      fixed_delay: 5s
      percentage: { numerator: 1, denominator: HUNDRED }

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Downstream timeouts must never exceed the remaining upstream budget; without deadline propagation, edge 504s trigger silent thread-pool exhaustion across backend hops.
  • 02Trade-off tip: factor: Relative Duration vs Epoch Timestamp; pro: Zero dependency on synchronized wall clocks.; con: Does not account for variable network queuing delay unless a small buffer is subtracted.
  • 03Watch out for: name: Orphaned DML Locks; description: Long UPDATE/DELETE statements ignore MAX_EXECUTION_TIME.; mitigation: Use Statement.cancel() via gRPC cancellation listener + innodb_lock_wait_timeout=5.
Behavioral Progression · Deadline Propagation Timeout Budgets

Deadline Propagation Timeout Budgets — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Downstream timeouts must never exceed the remaining upstream budget; without deadline propagation, edge 504s trigger silent thread-pool exhaustion across backend hops.

Synthesis / One connected design

Putting Deadline Propagation & Request Timeout Budgets Together

Downstream timeouts must never exceed the remaining upstream budget; without deadline propagation, edge 504s trigger silent thread-pool exhaustion across backend hops.

A candidate architecture, not a universal answer

Putting Deadline Propagation & Request Timeout Budgets Together

Step 1
Step 2
Step 3
  1. Parse incoming deadline duration into a monotonic System.nanoTime() deadline.
  2. Subtract 50ms hop overhead before making downstream gRPC/HTTP calls.
  3. Bind remaining ms to MySQL MAX_EXECUTION_TIME and skip retries if budget < 2 * p99.
Application-layer thread interruption does not automatically kill an in-flight MySQL query; you must bind the deadline to the SQL driver and DB engine.
Interview Readiness Check

Application-layer thread interruption does not automatically kill an in-flight MySQL query; you must bind the deadline to the SQL driver and DB engine.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Deadline Propagation & Request Timeout Budgets

Downstream timeouts must never exceed the remaining upstream budget; without deadline propagation, edge 504s trigger silent thread-pool exhaustion across backend hops.