During a mobile app push notification spike, `/v1/telemetry/events` jumps from 2,000 to 35,000 RPS on the shared API gateway and backend fleet (capacity 20,000 RPS). Naive global rate limiting and Tomcat queue saturation drop 42% of `/v1/checkout/confirm` (500 RPS) and `/v1/auth/token` (1,500 RPS) requests with HTTP 503.
From patterns to decisions.
During a mobile app push notification spike, `/v1/telemetry/events` jumps from 2,000 to 35,000 RPS on the shared API gateway and backend fleet (capacity 20,000 RPS). Naive global rate limiting and Tomcat queue saturation drop 42% of `/v1/checkout/confirm` (500 RPS) and `/v1/auth/token` (1,500 RPS) requests with HTTP 503.
Shared Tomcat FIFO queues and.
Shared Tomcat FIFO queues and priority-blind Envoy circuit breakers caused telemetry requests to saturate all 4,000 threads.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Shared pools maximize utilization at rest but fail catastrophically on skewed bursts. |
Step 1
Inspect Envoy upstream_rq_pending_overflow and Tomcat active thread metrics.
Step 2
Map traffic share by endpoint during the incident.
Alternative 1
Deploy telemetry on a separate Kubernetes deployment.
Failure Mode 1
FIFO head-of-line blocking.
upstream_rq_pending_overflow: 42% across all routes
Tag routes at Envoy, shed.
Tag routes at Envoy, shed Tier-3 at >75% load with 429 + Retry-After, and cap Tier-3 backend concurrency with a non-blocking Semaphore.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Telemetry experiences 429s earlier (at 75% load) to leave 25% headroom for Tier-0. |
Step 1
Strip external priority headers at ingress.
Step 2
Enforce Tier-3 Semaphore + systemLoadRatio check.
Alternative 1
Separate Ingress + separate backend deployments per tier.
Failure Mode 1
Semaphore leak if release() is not in a finally block.
if (tier == 3 && (systemLoadRatio() > 0.75 || !tier3Permits.tryAcquire())) { res.setStatus(429); return; }
A fixed 15,000 RPS limit.
A fixed 15,000 RPS limit still overloads the cluster if downstream DB latency triples and real capacity drops to 6,000 RPS.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Requires periodic min_rtt recalibration. |
Step 1
Compare static RPS vs Little's Law concurrency limit (L = lambda * W).
Step 2
Observe how gradient controllers shrink limits automatically as RTT rises.
Alternative 1
CPU/GC-based shedding.
Failure Mode 1
Static limits fail during brownouts.
gradient_controller_config:
min_rtt_calc_params:
interval: 60s
Offload Tier-3 shedding to an.
Offload Tier-3 shedding to an edge WAF / CDN rate-limiter or Envoy local_ratelimit filter before heavy JWT/body parsing.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Local rate limit is per-Envoy-pod rather than globally synchronized. |
Step 1
Place Envoy local_ratelimit early in the HTTP filter chain.
Step 2
Return 429 without invoking ext_authz or body buffering.
Alternative 1
Cloudflare / AWS WAF edge rule.
Failure Mode 1
Running ext_authz on sheddable traffic wastes auth service capacity.
name: envoy.filters.http.local_ratelimit
typed_config:
stat_prefix: telemetry_shed
Return randomized Retry-After headers (e.
Return randomized Retry-After headers (e.g., 20-40s) and configure SDKs with full-jitter exponential backoff + max-buffer drop.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Slight delay in analytics dashboards during traffic spikes. |
Step 1
Add random jitter to Retry-After on 429 responses.
Step 2
Cap SDK retry attempts at 3 before dropping non-critical telemetry.
Alternative 1
Client-side sampling (drop 50% at SDK when 429 rate > 5%).
Fixed Retry-After
10 creates synchronized 10s thundering herds.
res.setHeader("Retry-After", String.valueOf(20 + ThreadLocalRandom.current().nextInt(20)));
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Tag requests with criticality tiers at ingress, enforce priority-aware LIFO/adaptive load shedding, and isolate thread/connection pools via bulkheads.
- 02Trade-off tip: Requires periodic min_rtt recalibration.
- 03Watch out for: Running ext_authz on sheddable traffic wastes auth service capacity.
Priority Aware Load Shedding — Normal → Load / Fault → Recovery
Tag requests with criticality tiers at ingress, enforce priority-aware LIFO/adaptive load shedding, and isolate thread/connection pools via bulkheads.
Putting Priority-Aware Load Shedding Under Overload Together
Tag requests with criticality tiers at ingress, enforce priority-aware LIFO/adaptive load shedding, and isolate thread/connection pools via bulkheads.
Putting Priority-Aware Load Shedding Under Overload Together
- Strip external priority headers at ingress.
- Enforce Tier-3 Semaphore + systemLoadRatio check.
Interview Readiness Check
Offload Tier-3 shedding to an edge WAF / CDN rate-limiter or Envoy local_ratelimit filter before heavy JWT/body parsing.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Priority-Aware Load Shedding Under Overload
Tag requests with criticality tiers at ingress, enforce priority-aware LIFO/adaptive load shedding, and isolate thread/connection pools via bulkheads.
