Your Envoy ingress fleet (40 pods, 4 vCPU each) terminates 120,000 persistent mTLS/TLS 1.3 connections from 1,500 internal microservice pods. When an NLB target group or Envoy EDS/CDS reload resets connections simultaneously, all 1,500 pods immediately reconnect at ~60,000 handshakes/sec. Because TLS session tickets are disabled (`disable_stateless_session_resumption: true`) and leaf certs use RSA-2048, Envoy CPU pins at 100%, handshake latency hits 8 seconds, client timeouts trigger new reconnects, and the cluster enters a permanent handshake storm. Design a storm-proof TLS termination and reconnection architecture.
TL;DR30-second simplified summary▾
Your Envoy ingress fleet (40 pods, 4 vCPU each) terminates 120,000 persistent mTLS/TLS 1.3 connections from 1,500 internal microservice pods. When an NLB target group or Envoy EDS/CDS reload resets connections simultaneously, all 1,500 pods immediately reconnect at ~60,000 handshakes/sec. Because TLS session tickets are disabled (disable_stateless_session_resumption: true) and leaf certs use RSA-2048, Envoy CPU pins at 100%, handshake latency hits 8 seconds, client timeouts trigger new reconnects, and the cluster enters a permanent handshake storm. Design a storm-proof TLS termination and reconnection architecture.
Enable TLS 1.3 PSK session resumption with rotated shared STEKs, switch leaf certs from RSA-2048 to ECDSA P-256, and stagger reconnects with jitter + handshake rate limits.
From patterns to decisions.
Your Envoy ingress fleet (40 pods, 4 vCPU each) terminates 120,000 persistent mTLS/TLS 1.3 connections from 1,500 internal microservice pods. When an NLB target group or Envoy EDS/CDS reload resets connections simultaneously, all 1,500 pods immediately reconnect at ~60,000 handshakes/sec. Because TLS session tickets are disabled (`disable_stateless_session_resumption: true`) and leaf certs use RSA-2048, Envoy CPU pins at 100%, handshake latency hits 8 seconds, client timeouts trigger new reconnects, and the cluster enters a permanent handshake storm. Design a storm-proof TLS termination and reconnection architecture.
RSA-2048 private-key signing and ECDHE.
RSA-2048 private-key signing and ECDHE key exchange are computationally expensive asymmetric operations (~1 ms/handshake) compared to hardware-accelerated AES-GCM symmetric encryption.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Checking `ssl.session_reused / ssl.handshake` immediately reveals ticket misconfiguration. |
Step 1
Profile Envoy with perf top and observe RSA_private_decrypt / BN_mod_exp dominating CPU.
Step 2
Check envoy_ssl_handshake vs envoy_ssl_session_reused counters.
Step 3
Confirm session_reused == 0 due to disabled session tickets.
Alternative 1
Run openssl speed rsa2048 ecdsap256 on the target node to measure exact core capacity.
Failure Mode 1
Assuming scaling NLB/ALB fixes CPU when the bottleneck is Envoy's pod-local TLS termination.
curl -s localhost:9901/stats | grep -E 'ssl\.(handshake|session_reused|connection_error)'
Implementation & Guardrails (2).
Configure Envoy with SDS-backed session_ticket_keys_sds_secret_config, ECDSA P-256 certs, and jittered client backoff.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | ECDSA P-256 + shared STEK cuts storm CPU by >95% with zero hardware changes. |
Step 1
Issue ECDSA P-256 leaf certs from internal CA.
Step 2
Rotate a 2-key STEK Secret every 12h via CronJob/Vault.
Step 3
Add full jitter to client reconnects and max_connection_duration.
Alternative 1
Offload TLS termination to AWS ALB/NLB or hardware QAT cards.
Failure Mode 1
Rotating STEK by overwriting with a single key invalidates all existing tickets and causes a mini-storm.
common_http_protocol_options:
max_connection_duration: 300s
# Client-side jittered reconnect prevents synchronized 300s drops
No for general APIs.
No for general APIs: 0-RTT is vulnerable to replay attacks and is unnecessary because 1-RTT PSK already eliminates the asymmetric CPU bottleneck.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | 1-RTT PSK gives all the CPU savings with zero replay-attack surface. |
Step 1
Keep 0-RTT disabled on DownstreamTlsContext by default.
Step 2
Use 1-RTT PSK resumption to save CPU safely.
Step 3
Only consider 0-RTT on strictly idempotent GET CDN routes with anti-replay caches.
Alternative 1
If 0-RTT is required for mobile, reject non-GET methods with HTTP 425 Too Early.
Failure Mode 1
Allowing 0-RTT on /payments lets attackers replay payment requests.
# Omit enable_early_data (defaults to false) to prevent replay attacks on POST/PUT
With ECDSA P-256 (0.
With ECDSA P-256 (0.18 ms/handshake) and a 3-second client jitter window, even 0% resumption uses only ~7.2 cores (4.5% of fleet CPU).
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | ECDSA + jitter guarantees survival even on a 0% ticket cold start. |
Step 1
Set Envoy overload_manager to shed new connections if CPU/heap crosses 85%.
Step 2
Ensure client handshake timeout is >= 3 s so jittered queues drain without aborting.
Step 3
Warm critical gRPC channels during pod startupProbe.
Alternative 1
Use Envoy listener_filters rate limiting on TLS ClientHello.
Failure Mode 1
Short client handshake timeouts (e.g., 250 ms) turn a brief cold-start queue into an infinite retry loop.
overload_manager:
refresh_interval: 0.25s
resource_monitors:
- name: envoy.resource_monitors.fixed_heap
typed_config: { max_heap_size_bytes: 2147483648 }
Implementation & Guardrails (5).
Track envoy_ssl_session_reused / (envoy_ssl_handshake + envoy_ssl_session_reused) and test cross-pod resumption with openssl s_client.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Synthetic cross-pod verification catches broken SDS rotation before a storm happens. |
Step 1
Save a session ticket from Pod A using openssl s_client -sess_out ticket.pem.
Step 2
Connect to Pod B using -sess_in ticket.pem and verify Reused, TLSv1.3.
Step 3
Alert if production resumption ratio drops below 70%.
Alternative 1
Monitor ssl.session_reused per Envoy pod in Grafana.
Failure Mode 1
Testing against the NLB VIP might hit the same pod twice and give a false positive.
openssl s_client -connect ingress-pod-a:443 -tls1_3 -sess_out /tmp/sess.pem </dev/null
openssl s_client -connect ingress-pod-b:443 -tls1_3 -sess_in /tmp/sess.pem </dev/null | grep Reused
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Enable TLS 1.3 PSK session resumption with rotated shared STEKs, switch leaf certs from RSA-2048 to ECDSA P-256, and stagger reconnects with jitter + handshake rate limits.
- 02Trade-off tip: 1-RTT PSK gives all the CPU savings with zero replay-attack surface.
- 03Watch out for: Short client handshake timeouts (e.g., 250 ms) turn a brief cold-start queue into an infinite retry loop.
Putting Mitigating TLS Handshake Storms on Reconnects Together
Enable TLS 1.3 PSK session resumption with rotated shared STEKs, switch leaf certs from RSA-2048 to ECDSA P-256, and stagger reconnects with jitter + handshake rate limits.
Putting Mitigating TLS Handshake Storms on Reconnects Together
- Issue ECDSA P-256 leaf certs from internal CA.
- Rotate a 2-key STEK Secret every 12h via CronJob/Vault.
- Add full jitter to client reconnects and
max_connection_duration.
Interview Readiness Check
With ECDSA P-256 (0.18 ms/handshake) and a 3-second client jitter window, even 0% resumption uses only ~7.2 cores (4.5% of fleet CPU).
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Practice: Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Mitigating TLS Handshake Storms on Reconnects
Enable TLS 1.3 PSK session resumption with rotated shared STEKs, switch leaf certs from RSA-2048 to ECDSA P-256, and stagger reconnects with jitter + handshake rate limits.
