Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Priority-Aware Load Shedding
Under Overload.

During a mobile app push notification spike, /v1/telemetry/events jumps from 2,000 to 35,000 RPS on the shared API gateway and backend fleet (capacity 20,000 RPS). Naive global rate limiting and Tomcat queue saturation drop 42% of /v1/checkout/confirm (500 RPS) and /v1/auth/token (1,500 RPS) requests with HTTP 503.
Start the journey5 stages
Loading diagram…
Priority-Aware Load Shedding Under Overload architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

During a mobile app push notification spike, `/v1/telemetry/events` jumps from 2,000 to 35,000 RPS on the shared API gateway and backend fleet (capacity 20,000 RPS). Naive global rate limiting and Tomcat queue saturation drop 42% of `/v1/checkout/confirm` (500 RPS) and `/v1/auth/token` (1,500 RPS) requests with HTTP 503.

From patterns to decisions.

During a mobile app push notification spike, `/v1/telemetry/events` jumps from 2,000 to 35,000 RPS on the shared API gateway and backend fleet (capacity 20,000 RPS). Naive global rate limiting and Tomcat queue saturation drop 42% of `/v1/checkout/confirm` (500 RPS) and `/v1/auth/token` (1,500 RPS) requests with HTTP 503.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Shared Tomcat FIFO queues and.

Shared Tomcat FIFO queues and priority-blind Envoy circuit breakers caused telemetry requests to saturate all 4,000 threads.

ChoiceWhat it buysWhat it costs
Trade-off 1Shared pools maximize utilization at rest but fail catastrophically on skewed bursts.

Step 1

Inspect Envoy upstream_rq_pending_overflow and Tomcat active thread metrics.

Step 2

Map traffic share by endpoint during the incident.

Alternative 1

Deploy telemetry on a separate Kubernetes deployment.

Failure Mode 1

FIFO head-of-line blocking.

upstream_rq_pending_overflow: 42% across all routes
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Tag routes at Envoy, shed.

Tag routes at Envoy, shed Tier-3 at >75% load with 429 + Retry-After, and cap Tier-3 backend concurrency with a non-blocking Semaphore.

ChoiceWhat it buysWhat it costs
Trade-off 1Telemetry experiences 429s earlier (at 75% load) to leave 25% headroom for Tier-0.

Step 1

Strip external priority headers at ingress.

Step 2

Enforce Tier-3 Semaphore + systemLoadRatio check.

Alternative 1

Separate Ingress + separate backend deployments per tier.

Failure Mode 1

Semaphore leak if release() is not in a finally block.

if (tier == 3 && (systemLoadRatio() > 0.75 || !tier3Permits.tryAcquire())) { res.setStatus(429); return; }
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

A fixed 15,000 RPS limit.

A fixed 15,000 RPS limit still overloads the cluster if downstream DB latency triples and real capacity drops to 6,000 RPS.

ChoiceWhat it buysWhat it costs
Trade-off 1Requires periodic min_rtt recalibration.

Step 1

Compare static RPS vs Little's Law concurrency limit (L = lambda * W).

Step 2

Observe how gradient controllers shrink limits automatically as RTT rises.

Alternative 1

CPU/GC-based shedding.

Failure Mode 1

Static limits fail during brownouts.

gradient_controller_config:
  min_rtt_calc_params:
    interval: 60s
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Offload Tier-3 shedding to an.

Offload Tier-3 shedding to an edge WAF / CDN rate-limiter or Envoy local_ratelimit filter before heavy JWT/body parsing.

ChoiceWhat it buysWhat it costs
Trade-off 1Local rate limit is per-Envoy-pod rather than globally synchronized.

Step 1

Place Envoy local_ratelimit early in the HTTP filter chain.

Step 2

Return 429 without invoking ext_authz or body buffering.

Alternative 1

Cloudflare / AWS WAF edge rule.

Failure Mode 1

Running ext_authz on sheddable traffic wastes auth service capacity.

name: envoy.filters.http.local_ratelimit
typed_config:
  stat_prefix: telemetry_shed
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Return randomized Retry-After headers (e.

Return randomized Retry-After headers (e.g., 20-40s) and configure SDKs with full-jitter exponential backoff + max-buffer drop.

ChoiceWhat it buysWhat it costs
Trade-off 1Slight delay in analytics dashboards during traffic spikes.

Step 1

Add random jitter to Retry-After on 429 responses.

Step 2

Cap SDK retry attempts at 3 before dropping non-critical telemetry.

Alternative 1

Client-side sampling (drop 50% at SDK when 429 rate > 5%).

Fixed Retry-After

10 creates synchronized 10s thundering herds.

res.setHeader("Retry-After", String.valueOf(20 + ThreadLocalRandom.current().nextInt(20)));

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Tag requests with criticality tiers at ingress, enforce priority-aware LIFO/adaptive load shedding, and isolate thread/connection pools via bulkheads.
  • 02Trade-off tip: Requires periodic min_rtt recalibration.
  • 03Watch out for: Running ext_authz on sheddable traffic wastes auth service capacity.
Behavioral Progression · Priority Aware Load Shedding

Priority Aware Load Shedding — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Tag requests with criticality tiers at ingress, enforce priority-aware LIFO/adaptive load shedding, and isolate thread/connection pools via bulkheads.

Synthesis / One connected design

Putting Priority-Aware Load Shedding Under Overload Together

Tag requests with criticality tiers at ingress, enforce priority-aware LIFO/adaptive load shedding, and isolate thread/connection pools via bulkheads.

A candidate architecture, not a universal answer

Putting Priority-Aware Load Shedding Under Overload Together

Step 1
Step 2
  1. Strip external priority headers at ingress.
  2. Enforce Tier-3 Semaphore + systemLoadRatio check.
Offload Tier-3 shedding to an edge WAF / CDN rate-limiter or Envoy local_ratelimit filter before heavy JWT/body parsing.
Interview Readiness Check

Offload Tier-3 shedding to an edge WAF / CDN rate-limiter or Envoy local_ratelimit filter before heavy JWT/body parsing.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Priority-Aware Load Shedding Under Overload

Tag requests with criticality tiers at ingress, enforce priority-aware LIFO/adaptive load shedding, and isolate thread/connection pools via bulkheads.