Your payment-ledger gRPC service runs on Kubernetes behind an AWS Network Load Balancer (L4). During a 10x traffic surge, HPA scales the backend from 4 to 24 pods, yet CPU stays at 98% on the original 4 pods while the 20 new pods sit at 3% CPU. Existing Java and Python gRPC clients keep sending 100% of their 45,000 RPS over their initial subchannels. Diagnose why L4 load balancing fails for HTTP/2 and design a zero-downtime fix.
From patterns to decisions.
Your payment-ledger gRPC service runs on Kubernetes behind an AWS Network Load Balancer (L4). During a 10x traffic surge, HPA scales the backend from 4 to 24 pods, yet CPU stays at 98% on the original 4 pods while the 20 new pods sit at 3% CPU. Existing Java and Python gRPC clients keep sending 100% of their 45,000 RPS over their initial subchannels. Diagnose why L4 load balancing fails for HTTP/2 and design a zero-downtime fix.
L4 load balancers only route.
L4 load balancers only route on TCP connection establishment (SYN). Because HTTP/2 multiplexes all gRPC calls onto one long-lived TCP subchannel per VIP, existing clients never open new connections to hit the 20 new pods.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Headless DNS removes the L4 VIP bottleneck but requires clients to support round_robin LB. | |
| Trade-off 2 | Using an L7 Envoy proxy keeps a single VIP for external clients at the cost of +0.5 ms extra hop. |
Step 1
Inspect active TCP connections (ss -tnp | grep :9090) on old vs. new pods.
Step 2
Confirm grpc_server_started_total is 0 on newly scaled pods.
Step 3
Identify that ClusterIP / NLB VIP returns 1 IP, causing pick_first to hold 1 TCP socket indefinitely.
Alternative 1
Deploy Envoy or Linkerd/Istio L7 service mesh.
Alternative 2
Use gRPC-xDS control plane for dynamic endpoint push.
Failure Mode 1
Leaving default pick_first policy on headless DNS still pins all traffic to the first returned IP.
apiVersion: v1
kind: Service
metadata:
name: payment-ledger-headless
namespace: prod
spec:
clusterIP: None
selector:
app: payment-ledger
ports:
- name: grpc
port: 9090
targetPort: 9090
Combine headless DNS resolution with.
Combine headless DNS resolution with round_robin on the client and jittered maxConnectionAge + maxConnectionAgeGrace on the Netty server.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | 180s maxConnectionAge adds ~16 TLS handshakes/sec cluster-wide (negligible CPU). | |
| Trade-off 2 | Lower maxConnectionAge (e.g., 15s) speeds up rebalancing but increases TLS handshake CPU. |
Step 1
Set defaultLoadBalancingPolicy("round_robin") on ManagedChannelBuilder.
Step 2
Set maxConnectionAge to 180s with maxConnectionAgeGrace of 5s on NettyServerBuilder.
Step 3
In Python, pass options=[('grpc.lb_policy_name', 'round_robin')] to grpc.aio.insecure_channel / secure_channel.
Alternative 1
Envoy L7 proxy with lb_policy: LEAST_REQUEST for external clients.
Failure Mode 1
Omitting maxConnectionAgeGrace causes RST_STREAM on 2-second in-flight RPCs.
ManagedChannel channel = ManagedChannelBuilder
.forTarget("dns:///payment-ledger-headless.prod.svc.cluster.local:9090")
.defaultLoadBalancingPolicy("round_robin")
.build();
Server server = NettyServerBuilder.forPort(9090)
.maxConnectionAge(180, TimeUnit.SECONDS)
.maxConnectionAgeGrace(5, TimeUnit.SECONDS)
.addService(new LedgerServiceImpl())
.build();
Lowering MAX_CONNECTION_AGE to 10s behind.
Lowering MAX_CONNECTION_AGE to 10s behind an L4 NLB still leaves each client with only 1 subchannel—meaning any single client's burst still hits 1 pod, and TLS 1.3 handshakes spike 18x.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Client-side `round_robin` eliminates single-client hot-spotting without proxy infrastructure. | |
| Trade-off 2 | Requires internal DNS visibility to pod IPs (not possible across public internet). |
Compare per-client fan-out
1 subchannel behind VIP vs. 24 subchannels with headless DNS.
Calculate TLS 1.3 handshake rate
2,880 / 10s = 288 handshakes/sec vs. 16 handshakes/sec.
Step 3
Verify that round_robin balances every single RPC across all 24 pods immediately.
Alternative 1
AWS ALB (L7 gRPC target group) for external traffic.
Failure Mode 1
If grpc.dns_min_time_between_resolutions_ms is too high, newly scaled pods wait longer to receive traffic.
channel = grpc.aio.secure_channel(
"dns:///payment-ledger-headless.prod.svc.cluster.local:9090",
credentials,
options=[
("grpc.lb_policy_name", "round_robin"),
("grpc.dns_min_time_between_resolutions_ms", 5000),
],
)
At 200,000 sockets, full-mesh client-side.
At 200,000 sockets, full-mesh client-side LB wastes memory (~50 KB/socket = 10 GB) and causes epoll/TLS churn. Migrate to gRPC-xDS subsetting or an L7 Envoy gateway tier.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Envoy tier reduces total cluster sockets by 8x at the cost of ~0.4 ms p99 latency. | |
| Trade-off 2 | xDS subsetting keeps zero extra hops but requires running a control plane (Istio/Traffic Director). |
Step 1
Cap per-client subchannels using xDS subset load balancing (e.g., 16–24 endpoints per client).
Step 2
Alternatively, route through a stateless Envoy L7 proxy tier of 20 pods (1,000×20 + 20×200 = 24,000 sockets).
Alternative 1
Topology-aware routing (same-AZ preference) to shrink cross-AZ connection meshes.
Failure Mode 1
Without subsetting, 1,000 clients × 200 servers exhausts conntrack tables and ephemeral ports.
# Envoy L7 upstream cluster with bounded connections
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: LEAST_REQUEST
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options:
max_concurrent_streams: 250
Kubernetes removes the pod from.
Kubernetes removes the pod from Endpoints asynchronously while sending SIGTERM. Without a preStop hook and GOAWAY grace period, the pod closes sockets before kube-proxy/CoreDNS updates propagate.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Adds 5 seconds to pod rollout time per pod. | |
| Trade-off 2 | Eliminates race-condition UNAVAILABLE errors during EndpointSlice propagation. |
Step 1
Add lifecycle.preStop.exec (sleep 5) so the pod stays alive while CoreDNS/EndpointSlices propagate.
Step 2
Call server.shutdown() and awaitTermination(20, SECONDS) on SIGTERM to emit HTTP/2 GOAWAY.
Alternative 1
Enable client-side gRPC retryPolicy for idempotent unary reads on UNAVAILABLE.
Failure Mode 1
Killing the process before sleep 5 drops streams still in flight from clients with cached DNS.
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 30
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Solve HTTP/2 pinning by switching to client-side L7 round-robin over headless DNS, inserting an L7 Envoy proxy, or enforcing jittered server-side MAX_CONNECTION_AGE.
- 02Trade-off tip: Client-side
round_robineliminates single-client hot-spotting without proxy infrastructure. - 03Trade-off tip: Requires internal DNS visibility to pod IPs (not possible across public internet).
- 04Watch out for: Without subsetting, 1,000 clients × 200 servers exhausts conntrack tables and ephemeral ports.
Grpc Http2 L7 Load Balancing — Normal → Load / Fault → Recovery
Solve HTTP/2 pinning by switching to client-side L7 round-robin over headless DNS, inserting an L7 Envoy proxy, or enforcing jittered server-side MAXCONNECTIONAGE.
Putting Load Balancing gRPC & HTTP/2 Across Kubernetes Pods Together
Solve HTTP/2 pinning by switching to client-side L7 round-robin over headless DNS, inserting an L7 Envoy proxy, or enforcing jittered server-side MAX_CONNECTION_AGE.
Putting Load Balancing gRPC & HTTP/2 Across Kubernetes Pods Together
- Set
defaultLoadBalancingPolicy("round_robin")onManagedChannelBuilder. - Set
maxConnectionAgeto 180s withmaxConnectionAgeGraceof 5s onNettyServerBuilder. - In Python, pass
options=[('grpc.lb_policy_name', 'round_robin')]togrpc.aio.insecure_channel/secure_channel.
Interview Readiness Check
At 200,000 sockets, full-mesh client-side LB wastes memory (~50 KB/socket = 10 GB) and causes epoll/TLS churn. Migrate to gRPC-xDS subsetting or an L7 Envoy gateway tier.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Load Balancing gRPC & HTTP/2 Across Kubernetes Pods
Solve HTTP/2 pinning by switching to client-side L7 round-robin over headless DNS, inserting an L7 Envoy proxy, or enforcing jittered server-side MAX_CONNECTION_AGE.
