Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Mitigating TLS Handshake
Storms on Reconnects.

Your Envoy ingress fleet (40 pods, 4 vCPU each) terminates 120,000 persistent mTLS/TLS 1.3 connections from 1,500 internal microservice pods. When an NLB target group or Envoy EDS/CDS reload resets connections simultaneously, all 1,500 pods immediately reconnect at ~60,000 handshakes/sec. Because TLS session tickets are disabled (disable_stateless_session_resumption: true) and leaf certs use RSA-2048, Envoy CPU pins at 100%, handshake latency hits 8 seconds, client timeouts trigger new reconnects, and the cluster enters a permanent handshake storm. Design a storm-proof TLS termination and reconnection architecture.
Start the journey5 stages
Loading diagram…
Mitigating TLS Handshake Storms on Reconnects architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your Envoy ingress fleet (40 pods, 4 vCPU each) terminates 120,000 persistent mTLS/TLS 1.3 connections from 1,500 internal microservice pods. When an NLB target group or Envoy EDS/CDS reload resets connections simultaneously, all 1,500 pods immediately reconnect at ~60,000 handshakes/sec. Because TLS session tickets are disabled (`disable_stateless_session_resumption: true`) and leaf certs use RSA-2048, Envoy CPU pins at 100%, handshake latency hits 8 seconds, client timeouts trigger new reconnects, and the cluster enters a permanent handshake storm. Design a storm-proof TLS termination and reconnection architecture.

TL;DR30-second simplified summary
▾

Your Envoy ingress fleet (40 pods, 4 vCPU each) terminates 120,000 persistent mTLS/TLS 1.3 connections from 1,500 internal microservice pods. When an NLB target group or Envoy EDS/CDS reload resets connections simultaneously, all 1,500 pods immediately reconnect at ~60,000 handshakes/sec. Because TLS session tickets are disabled (disable_stateless_session_resumption: true) and leaf certs use RSA-2048, Envoy CPU pins at 100%, handshake latency hits 8 seconds, client timeouts trigger new reconnects, and the cluster enters a permanent handshake storm. Design a storm-proof TLS termination and reconnection architecture.

Enable TLS 1.3 PSK session resumption with rotated shared STEKs, switch leaf certs from RSA-2048 to ECDSA P-256, and stagger reconnects with jitter + handshake rate limits.

From patterns to decisions.

Your Envoy ingress fleet (40 pods, 4 vCPU each) terminates 120,000 persistent mTLS/TLS 1.3 connections from 1,500 internal microservice pods. When an NLB target group or Envoy EDS/CDS reload resets connections simultaneously, all 1,500 pods immediately reconnect at ~60,000 handshakes/sec. Because TLS session tickets are disabled (`disable_stateless_session_resumption: true`) and leaf certs use RSA-2048, Envoy CPU pins at 100%, handshake latency hits 8 seconds, client timeouts trigger new reconnects, and the cluster enters a permanent handshake storm. Design a storm-proof TLS termination and reconnection architecture.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

RSA-2048 private-key signing and ECDHE.

RSA-2048 private-key signing and ECDHE key exchange are computationally expensive asymmetric operations (~1 ms/handshake) compared to hardware-accelerated AES-GCM symmetric encryption.

ChoiceWhat it buysWhat it costs
Trade-off 1Checking `ssl.session_reused / ssl.handshake` immediately reveals ticket misconfiguration.

Step 1

Profile Envoy with perf top and observe RSA_private_decrypt / BN_mod_exp dominating CPU.

Step 2

Check envoy_ssl_handshake vs envoy_ssl_session_reused counters.

Step 3

Confirm session_reused == 0 due to disabled session tickets.

Alternative 1

Run openssl speed rsa2048 ecdsap256 on the target node to measure exact core capacity.

Failure Mode 1

Assuming scaling NLB/ALB fixes CPU when the bottleneck is Envoy's pod-local TLS termination.

curl -s localhost:9901/stats | grep -E 'ssl\.(handshake|session_reused|connection_error)'
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Implementation & Guardrails (2).

Configure Envoy with SDS-backed session_ticket_keys_sds_secret_config, ECDSA P-256 certs, and jittered client backoff.

ChoiceWhat it buysWhat it costs
Trade-off 1ECDSA P-256 + shared STEK cuts storm CPU by >95% with zero hardware changes.

Step 1

Issue ECDSA P-256 leaf certs from internal CA.

Step 2

Rotate a 2-key STEK Secret every 12h via CronJob/Vault.

Step 3

Add full jitter to client reconnects and max_connection_duration.

Alternative 1

Offload TLS termination to AWS ALB/NLB or hardware QAT cards.

Failure Mode 1

Rotating STEK by overwriting with a single key invalidates all existing tickets and causes a mini-storm.

common_http_protocol_options:
  max_connection_duration: 300s
  # Client-side jittered reconnect prevents synchronized 300s drops
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

No for general APIs.

No for general APIs: 0-RTT is vulnerable to replay attacks and is unnecessary because 1-RTT PSK already eliminates the asymmetric CPU bottleneck.

ChoiceWhat it buysWhat it costs
Trade-off 11-RTT PSK gives all the CPU savings with zero replay-attack surface.

Step 1

Keep 0-RTT disabled on DownstreamTlsContext by default.

Step 2

Use 1-RTT PSK resumption to save CPU safely.

Step 3

Only consider 0-RTT on strictly idempotent GET CDN routes with anti-replay caches.

Alternative 1

If 0-RTT is required for mobile, reject non-GET methods with HTTP 425 Too Early.

Failure Mode 1

Allowing 0-RTT on /payments lets attackers replay payment requests.

# Omit enable_early_data (defaults to false) to prevent replay attacks on POST/PUT
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

With ECDSA P-256 (0.

With ECDSA P-256 (0.18 ms/handshake) and a 3-second client jitter window, even 0% resumption uses only ~7.2 cores (4.5% of fleet CPU).

ChoiceWhat it buysWhat it costs
Trade-off 1ECDSA + jitter guarantees survival even on a 0% ticket cold start.

Step 1

Set Envoy overload_manager to shed new connections if CPU/heap crosses 85%.

Step 2

Ensure client handshake timeout is >= 3 s so jittered queues drain without aborting.

Step 3

Warm critical gRPC channels during pod startupProbe.

Alternative 1

Use Envoy listener_filters rate limiting on TLS ClientHello.

Failure Mode 1

Short client handshake timeouts (e.g., 250 ms) turn a brief cold-start queue into an infinite retry loop.

overload_manager:
  refresh_interval: 0.25s
  resource_monitors:
    - name: envoy.resource_monitors.fixed_heap
      typed_config: { max_heap_size_bytes: 2147483648 }
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Implementation & Guardrails (5).

Track envoy_ssl_session_reused / (envoy_ssl_handshake + envoy_ssl_session_reused) and test cross-pod resumption with openssl s_client.

ChoiceWhat it buysWhat it costs
Trade-off 1Synthetic cross-pod verification catches broken SDS rotation before a storm happens.

Step 1

Save a session ticket from Pod A using openssl s_client -sess_out ticket.pem.

Step 2

Connect to Pod B using -sess_in ticket.pem and verify Reused, TLSv1.3.

Step 3

Alert if production resumption ratio drops below 70%.

Alternative 1

Monitor ssl.session_reused per Envoy pod in Grafana.

Failure Mode 1

Testing against the NLB VIP might hit the same pod twice and give a false positive.

openssl s_client -connect ingress-pod-a:443 -tls1_3 -sess_out /tmp/sess.pem </dev/null
openssl s_client -connect ingress-pod-b:443 -tls1_3 -sess_in /tmp/sess.pem </dev/null | grep Reused

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Enable TLS 1.3 PSK session resumption with rotated shared STEKs, switch leaf certs from RSA-2048 to ECDSA P-256, and stagger reconnects with jitter + handshake rate limits.
  • 02Trade-off tip: 1-RTT PSK gives all the CPU savings with zero replay-attack surface.
  • 03Watch out for: Short client handshake timeouts (e.g., 250 ms) turn a brief cold-start queue into an infinite retry loop.
Put it together · Synthesis

Putting Mitigating TLS Handshake Storms on Reconnects Together

Enable TLS 1.3 PSK session resumption with rotated shared STEKs, switch leaf certs from RSA-2048 to ECDSA P-256, and stagger reconnects with jitter + handshake rate limits.

A candidate architecture, not a universal answer

Putting Mitigating TLS Handshake Storms on Reconnects Together

Step 1
Step 2
Step 3
  1. Issue ECDSA P-256 leaf certs from internal CA.
  2. Rotate a 2-key STEK Secret every 12h via CronJob/Vault.
  3. Add full jitter to client reconnects and max_connection_duration.
With ECDSA P-256 (0.18 ms/handshake) and a 3-second client jitter window, even 0% resumption uses only ~7.2 cores (4.5% of fleet CPU).
Interview Readiness Check

With ECDSA P-256 (0.18 ms/handshake) and a 3-second client jitter window, even 0% resumption uses only ~7.2 cores (4.5% of fleet CPU).

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Mitigating TLS Handshake Storms on Reconnects

Enable TLS 1.3 PSK session resumption with rotated shared STEKs, switch leaf certs from RSA-2048 to ECDSA P-256, and stagger reconnects with jitter + handshake rate limits.