Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Load Balancing gRPC &
HTTP/2 Across Kubernetes Pods.

Your payment-ledger gRPC service runs on Kubernetes behind an AWS Network Load Balancer (L4). During a 10x traffic surge, HPA scales the backend from 4 to 24 pods, yet CPU stays at 98% on the original 4 pods while the 20 new pods sit at 3% CPU. Existing Java and Python gRPC clients keep sending 100% of their 45,000 RPS over their initial subchannels. Diagnose why L4 load balancing fails for HTTP/2 and design a zero-downtime fix.
Start the journey5 stages
Loading diagram…
Load Balancing gRPC & HTTP/2 Across Kubernetes Pods architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your payment-ledger gRPC service runs on Kubernetes behind an AWS Network Load Balancer (L4). During a 10x traffic surge, HPA scales the backend from 4 to 24 pods, yet CPU stays at 98% on the original 4 pods while the 20 new pods sit at 3% CPU. Existing Java and Python gRPC clients keep sending 100% of their 45,000 RPS over their initial subchannels. Diagnose why L4 load balancing fails for HTTP/2 and design a zero-downtime fix.

From patterns to decisions.

Your payment-ledger gRPC service runs on Kubernetes behind an AWS Network Load Balancer (L4). During a 10x traffic surge, HPA scales the backend from 4 to 24 pods, yet CPU stays at 98% on the original 4 pods while the 20 new pods sit at 3% CPU. Existing Java and Python gRPC clients keep sending 100% of their 45,000 RPS over their initial subchannels. Diagnose why L4 load balancing fails for HTTP/2 and design a zero-downtime fix.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

L4 load balancers only route.

L4 load balancers only route on TCP connection establishment (SYN). Because HTTP/2 multiplexes all gRPC calls onto one long-lived TCP subchannel per VIP, existing clients never open new connections to hit the 20 new pods.

ChoiceWhat it buysWhat it costs
Trade-off 1Headless DNS removes the L4 VIP bottleneck but requires clients to support round_robin LB.
Trade-off 2Using an L7 Envoy proxy keeps a single VIP for external clients at the cost of +0.5 ms extra hop.

Step 1

Inspect active TCP connections (ss -tnp | grep :9090) on old vs. new pods.

Step 2

Confirm grpc_server_started_total is 0 on newly scaled pods.

Step 3

Identify that ClusterIP / NLB VIP returns 1 IP, causing pick_first to hold 1 TCP socket indefinitely.

Alternative 1

Deploy Envoy or Linkerd/Istio L7 service mesh.

Alternative 2

Use gRPC-xDS control plane for dynamic endpoint push.

Failure Mode 1

Leaving default pick_first policy on headless DNS still pins all traffic to the first returned IP.

apiVersion: v1
kind: Service
metadata:
  name: payment-ledger-headless
  namespace: prod
spec:
  clusterIP: None
  selector:
    app: payment-ledger
  ports:
    - name: grpc
      port: 9090
      targetPort: 9090
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Combine headless DNS resolution with.

Combine headless DNS resolution with round_robin on the client and jittered maxConnectionAge + maxConnectionAgeGrace on the Netty server.

ChoiceWhat it buysWhat it costs
Trade-off 1180s maxConnectionAge adds ~16 TLS handshakes/sec cluster-wide (negligible CPU).
Trade-off 2Lower maxConnectionAge (e.g., 15s) speeds up rebalancing but increases TLS handshake CPU.

Step 1

Set defaultLoadBalancingPolicy("round_robin") on ManagedChannelBuilder.

Step 2

Set maxConnectionAge to 180s with maxConnectionAgeGrace of 5s on NettyServerBuilder.

Step 3

In Python, pass options=[('grpc.lb_policy_name', 'round_robin')] to grpc.aio.insecure_channel / secure_channel.

Alternative 1

Envoy L7 proxy with lb_policy: LEAST_REQUEST for external clients.

Failure Mode 1

Omitting maxConnectionAgeGrace causes RST_STREAM on 2-second in-flight RPCs.

ManagedChannel channel = ManagedChannelBuilder
    .forTarget("dns:///payment-ledger-headless.prod.svc.cluster.local:9090")
    .defaultLoadBalancingPolicy("round_robin")
    .build();

Server server = NettyServerBuilder.forPort(9090)
    .maxConnectionAge(180, TimeUnit.SECONDS)
    .maxConnectionAgeGrace(5, TimeUnit.SECONDS)
    .addService(new LedgerServiceImpl())
    .build();
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

Lowering MAX_CONNECTION_AGE to 10s behind.

Lowering MAX_CONNECTION_AGE to 10s behind an L4 NLB still leaves each client with only 1 subchannel—meaning any single client's burst still hits 1 pod, and TLS 1.3 handshakes spike 18x.

ChoiceWhat it buysWhat it costs
Trade-off 1Client-side `round_robin` eliminates single-client hot-spotting without proxy infrastructure.
Trade-off 2Requires internal DNS visibility to pod IPs (not possible across public internet).

Compare per-client fan-out

1 subchannel behind VIP vs. 24 subchannels with headless DNS.

Calculate TLS 1.3 handshake rate

2,880 / 10s = 288 handshakes/sec vs. 16 handshakes/sec.

Step 3

Verify that round_robin balances every single RPC across all 24 pods immediately.

Alternative 1

AWS ALB (L7 gRPC target group) for external traffic.

Failure Mode 1

If grpc.dns_min_time_between_resolutions_ms is too high, newly scaled pods wait longer to receive traffic.

channel = grpc.aio.secure_channel(
    "dns:///payment-ledger-headless.prod.svc.cluster.local:9090",
    credentials,
    options=[
        ("grpc.lb_policy_name", "round_robin"),
        ("grpc.dns_min_time_between_resolutions_ms", 5000),
    ],
)
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

At 200,000 sockets, full-mesh client-side.

At 200,000 sockets, full-mesh client-side LB wastes memory (~50 KB/socket = 10 GB) and causes epoll/TLS churn. Migrate to gRPC-xDS subsetting or an L7 Envoy gateway tier.

ChoiceWhat it buysWhat it costs
Trade-off 1Envoy tier reduces total cluster sockets by 8x at the cost of ~0.4 ms p99 latency.
Trade-off 2xDS subsetting keeps zero extra hops but requires running a control plane (Istio/Traffic Director).

Step 1

Cap per-client subchannels using xDS subset load balancing (e.g., 16–24 endpoints per client).

Step 2

Alternatively, route through a stateless Envoy L7 proxy tier of 20 pods (1,000×20 + 20×200 = 24,000 sockets).

Alternative 1

Topology-aware routing (same-AZ preference) to shrink cross-AZ connection meshes.

Failure Mode 1

Without subsetting, 1,000 clients × 200 servers exhausts conntrack tables and ephemeral ports.

# Envoy L7 upstream cluster with bounded connections
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: LEAST_REQUEST
typed_extension_protocol_options:
  envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
    "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
    explicit_http_config:
      http2_protocol_options:
        max_concurrent_streams: 250
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Kubernetes removes the pod from.

Kubernetes removes the pod from Endpoints asynchronously while sending SIGTERM. Without a preStop hook and GOAWAY grace period, the pod closes sockets before kube-proxy/CoreDNS updates propagate.

ChoiceWhat it buysWhat it costs
Trade-off 1Adds 5 seconds to pod rollout time per pod.
Trade-off 2Eliminates race-condition UNAVAILABLE errors during EndpointSlice propagation.

Step 1

Add lifecycle.preStop.exec (sleep 5) so the pod stays alive while CoreDNS/EndpointSlices propagate.

Step 2

Call server.shutdown() and awaitTermination(20, SECONDS) on SIGTERM to emit HTTP/2 GOAWAY.

Alternative 1

Enable client-side gRPC retryPolicy for idempotent unary reads on UNAVAILABLE.

Failure Mode 1

Killing the process before sleep 5 drops streams still in flight from clients with cached DNS.

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 30

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Solve HTTP/2 pinning by switching to client-side L7 round-robin over headless DNS, inserting an L7 Envoy proxy, or enforcing jittered server-side MAX_CONNECTION_AGE.
  • 02Trade-off tip: Client-side round_robin eliminates single-client hot-spotting without proxy infrastructure.
  • 03Trade-off tip: Requires internal DNS visibility to pod IPs (not possible across public internet).
  • 04Watch out for: Without subsetting, 1,000 clients × 200 servers exhausts conntrack tables and ephemeral ports.
Behavioral Progression · Grpc Http2 L7 Load Balancing

Grpc Http2 L7 Load Balancing — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Solve HTTP/2 pinning by switching to client-side L7 round-robin over headless DNS, inserting an L7 Envoy proxy, or enforcing jittered server-side MAXCONNECTIONAGE.

Synthesis / One connected design

Putting Load Balancing gRPC & HTTP/2 Across Kubernetes Pods Together

Solve HTTP/2 pinning by switching to client-side L7 round-robin over headless DNS, inserting an L7 Envoy proxy, or enforcing jittered server-side MAX_CONNECTION_AGE.

A candidate architecture, not a universal answer

Putting Load Balancing gRPC & HTTP/2 Across Kubernetes Pods Together

Step 1
Step 2
Step 3
  1. Set defaultLoadBalancingPolicy("round_robin") on ManagedChannelBuilder.
  2. Set maxConnectionAge to 180s with maxConnectionAgeGrace of 5s on NettyServerBuilder.
  3. In Python, pass options=[('grpc.lb_policy_name', 'round_robin')] to grpc.aio.insecure_channel / secure_channel.
At 200,000 sockets, full-mesh client-side LB wastes memory (~50 KB/socket = 10 GB) and causes epoll/TLS churn. Migrate to gRPC-xDS subsetting or an L7 Envoy gateway tier.
Interview Readiness Check

At 200,000 sockets, full-mesh client-side LB wastes memory (~50 KB/socket = 10 GB) and causes epoll/TLS churn. Migrate to gRPC-xDS subsetting or an L7 Envoy gateway tier.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Load Balancing gRPC & HTTP/2 Across Kubernetes Pods

Solve HTTP/2 pinning by switching to client-side L7 round-robin over headless DNS, inserting an L7 Envoy proxy, or enforcing jittered server-side MAX_CONNECTION_AGE.