Every time a high-throughput API gateway / payment service (20,000 RPS across 40 Kubernetes pods behind an AWS ALB and Envoy ingress) runs a rolling deployment, clients observe a spike of 502 Bad Gateway errors and TCP RSTs for 15–30 seconds. Investigation shows pods immediately close Netty/Tomcat listener sockets upon receiving SIGTERM while kube-proxy and Envoy still route new and keep-alive requests to the terminating pod IP. Design a zero-downtime rolling deployment and graceful shutdown architecture.
From patterns to decisions.
Every time a high-throughput API gateway / payment service (20,000 RPS across 40 Kubernetes pods behind an AWS ALB and Envoy ingress) runs a rolling deployment, clients observe a spike of 502 Bad Gateway errors and TCP RSTs for 15–30 seconds. Investigation shows pods immediately close Netty/Tomcat listener sockets upon receiving SIGTERM while kube-proxy and Envoy still route new and keep-alive requests to the terminating pod IP. Design a zero-downtime rolling deployment and graceful shutdown architecture.
Implementation & Guardrails (1).
Kubernetes triggers EndpointSlice removal and container SIGTERM concurrently. Because the JVM closes port 8080 in ~15 ms while Envoy/kube-proxy take 2-8s to update routing tables, thousands of requests hit a closed socket.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Relying only on readinessProbe failure does not help once SIGTERM is sent, because kubelet doesn't wait for readinessProbe during deletion. | |
| Trade-off 2 | Shortening readinessProbe periodSeconds increases API server load without fixing the t=0 race. |
Trace t=0
Pod enters Terminating state.
Step 2
Compare control-plane propagation (API server -> EndpointSlice -> kube-proxy/Envoy/ALB = 2-8s) vs. JVM socket close (15 ms).
Step 3
Identify that HTTP/1.1 keep-alive connections already established by Envoy also fail if closed abruptly.
Alternative 1
Use AWS ALB TargetGroup IP mode with Pod Readiness Gates + preStop.
Alternative 2
Use Envoy graceful shutdown via Istio sidecar terminationDrainDuration.
Failure Mode 1
17,500+ dropped requests per deployment wave at 20,000 RPS.
Failure Mode 2
TCP RSTs on reused HTTP/1.1 keep-alive connections.
# Broken default configuration causing 502s
spec:
terminationGracePeriodSeconds: 30
containers:
- name: pricing-api
# Missing lifecycle.preStop -> SIGTERM arrives immediately at t=0
Pair maxUnavailable.
Pair maxUnavailable: 0, preStop sleep 10, and terminationGracePeriodSeconds: 45 with Spring Boot server.shutdown=graceful and 65s keep-alive alignment.
| Choice | What it buys | What it costs |
|---|---|---|
| maxSurge | 25% requires 10 extra pods worth of cluster capacity during rollouts. | |
| Trade-off 2 | 10s preStop adds ~40s to a 4-wave deployment. |
Step 1
Set strategy.rollingUpdate.maxUnavailable=0 and maxSurge=25%.
Step 2
Add preStop sleep 10 and terminationGracePeriodSeconds: 45.
Step 3
Enable server.shutdown=graceful and set timeout-per-shutdown-phase=30s.
Alternative 1
Use Blue/Green deployment with ALB weighted target groups for instant rollback.
Alternative 2
Use Argo Rollouts for automated canary analysis before scaling down old pods.
Failure Mode 1
If terminationGracePeriodSeconds is left at 30s while preStop is 10s, graceful drain only gets 20s before SIGKILL.
Failure Mode 2
Shell-form Docker ENTRYPOINT swallows SIGTERM, causing a 45s hang followed by SIGKILL.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-gateway
spec:
replicas: 40
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 25%
template:
spec:
terminationGracePeriodSeconds: 45
containers:
- name: payment-gateway
image: registry.internal/payment-gateway:2.4.1
ports:
- containerPort: 8080
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 10"]
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
periodSeconds: 5
failureThreshold: 2
If the server closes an.
If the server closes an idle HTTP/1.1 connection before the ALB/Envoy does, the proxy may dispatch a request into a closing socket causing a 502. Setting server idle timeout to 65s (>60s ALB) ensures the proxy closes first.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Longer server keep-alive (65s) holds slightly more file descriptors per pod. | |
| Trade-off 2 | HTTP/2 multiplexing avoids keep-alive races via GOAWAY but requires TLS/ALPN or h2c between Envoy and pod. |
Step 1
Check upstream ALB/Envoy idle_timeout (typically 60s).
Step 2
Configure Spring Boot Tomcat/Netty keepAliveTimeout to 65s.
Step 3
Verify HTTP/2 server sends GOAWAY on SIGTERM so Envoy opens new streams on healthy pods.
Alternative 1
Disable keep-alive between Envoy and pods (adds TCP handshake latency).
Alternative 2
Configure Envoy max_connection_duration with jitter.
Failure Mode 1
Reverse inequality (server 55s < ALB 60s) produces intermittent 502s even outside deployments.
Failure Mode 2
Ignoring Envoy circuit-breaker max_connections on surge pods.
# application.yaml (Spring Boot 3)
server:
shutdown: graceful
tomcat:
keep-alive-timeout: 65s
max-keep-alive-requests: 10000
spring:
lifecycle:
timeout-per-shutdown-phase: 30s
management:
endpoint:
health:
probes:
enabled: true
Long-lived WebSocket/SSE streams will be.
Long-lived WebSocket/SSE streams will be force-closed at 45s unless the application handles SIGTERM by sending a reconnect/close frame with jittered backoff.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Active WebSocket close triggers immediate client reconnects, requiring jitter to avoid a thundering herd on new pods. | |
| Trade-off 2 | Extending terminationGracePeriodSeconds to 3600s for WebSockets blocks CI/CD rollouts for an hour. |
Step 1
Listen for ContextClosedEvent in Java to notify active WebSocket/SSE clients.
Step 2
Send WebSocket CloseStatus.SERVICE_RESTARTED (1012) so clients reconnect with 1-5s jitter.
Step 3
Cap non-streaming HTTP requests with a 25s server timeout.
Alternative 1
Separate stateless REST pods from stateful WebSocket gateway pods.
Alternative 2
Use NATS/Redis Streams to replay missed messages after client reconnect.
Failure Mode 1
Reconnection storm if all 50,000 WebSocket clients reconnect simultaneously without jitter.
Failure Mode 2
Kubelet SIGKILL leaving half-open TCP sockets on clients without TCP keepalive.
@Component
public class GracefulWebSocketDrainer implements ApplicationListener<ContextClosedEvent> {
private final SessionRegistry sessionRegistry;
public GracefulWebSocketDrainer(SessionRegistry sessionRegistry) {
this.sessionRegistry = sessionRegistry;
}
@Override
public void onApplicationEvent(ContextClosedEvent event) {
// Instruct clients to reconnect to healthy surge pods with jitter
sessionRegistry.activeSessions().forEach(session ->
session.closeQuietly(CloseStatus.SERVICE_RESTARTED));
}
}
Pair the Deployment with a.
Pair the Deployment with a PodDisruptionBudget (minAvailable: 85%) so voluntary node drains respect the same zero-downtime capacity floor as rolling updates.
| Choice | What it buys | What it costs |
|---|---|---|
| Setting minAvailable | 100% blocks all node upgrades and drains indefinitely. | |
| Trade-off 2 | 85% allows 6 pods to drain concurrently while preserving enough capacity. |
Step 1
Define a PodDisruptionBudget with minAvailable: 85% (or maxUnavailable: 15%).
Step 2
Ensure Kubernetes Cluster Autoscaler / Karpenter respects PDBs during consolidation.
Step 3
Verify preStop + graceful shutdown also runs on kubelet eviction/drain.
Alternative 1
Use maxUnavailable: 1 for ultra-critical small deployments (<10 replicas).
Alternative 2
Schedule node drains during off-peak hours via Karpenter Disruption Budgets.
Failure Mode 1
PDB deadlock if replicas=1 and minAvailable=1.
Failure Mode 2
Spot instance 2-minute interruption notice bypassing normal PDB if node-termination-handler is missing.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: payment-gateway-pdb
spec:
minAvailable: 85%
selector:
matchLabels:
app: payment-gateway
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Kubernetes sends SIGTERM to the container in parallel with removing the pod from EndpointSlices; without a preStop delay, the app closes its listening socket before load balancers stop routing traffic to it.
- 02Trade-off tip: Longer server keep-alive (65s) holds slightly more file descriptors per pod.
- 03Trade-off tip: HTTP/2 multiplexing avoids keep-alive races via GOAWAY but requires TLS/ALPN or h2c between Envoy and pod.
- 04Watch out for: Reconnection storm if all 50,000 WebSocket clients reconnect simultaneously without jitter.
- 05Watch out for: Kubelet SIGKILL leaving half-open TCP sockets on clients without TCP keepalive.
Graceful Shutdown Rolling Deployments — Normal → Load / Fault → Recovery
Kubernetes sends SIGTERM to the container in parallel with removing the pod from EndpointSlices; without a preStop delay, the app closes its listening socket before load balancers stop routing traffic to it.
Putting Graceful Shutdown in Rolling Deployments Together
Kubernetes sends SIGTERM to the container in parallel with removing the pod from EndpointSlices; without a preStop delay, the app closes its listening socket before load balancers stop routing traffic to it.
Putting Graceful Shutdown in Rolling Deployments Together
- Set strategy.rollingUpdate.maxUnavailable=0 and maxSurge=25%.
- Add preStop sleep 10 and terminationGracePeriodSeconds: 45.
- Enable server.shutdown=graceful and set timeout-per-shutdown-phase=30s.
Interview Readiness Check
Long-lived WebSocket/SSE streams will be force-closed at 45s unless the application handles SIGTERM by sending a reconnect/close frame with jittered backoff.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Graceful Shutdown in Rolling Deployments
Kubernetes sends SIGTERM to the container in parallel with removing the pod from EndpointSlices; without a preStop delay, the app closes its listening socket before load balancers stop routing traffic to it.
