Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Graceful Shutdown in
Rolling Deployments.

Every time a high-throughput API gateway / payment service (20,000 RPS across 40 Kubernetes pods behind an AWS ALB and Envoy ingress) runs a rolling deployment, clients observe a spike of 502 Bad Gateway errors and TCP RSTs for 15–30 seconds. Investigation shows pods immediately close Netty/Tomcat listener sockets upon receiving SIGTERM while kube-proxy and Envoy still route new and keep-alive requests to the terminating pod IP. Design a zero-downtime rolling deployment and graceful shutdown architecture.
Start the journey5 stages
Loading diagram…
Graceful Shutdown in Rolling Deployments architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Every time a high-throughput API gateway / payment service (20,000 RPS across 40 Kubernetes pods behind an AWS ALB and Envoy ingress) runs a rolling deployment, clients observe a spike of 502 Bad Gateway errors and TCP RSTs for 15–30 seconds. Investigation shows pods immediately close Netty/Tomcat listener sockets upon receiving SIGTERM while kube-proxy and Envoy still route new and keep-alive requests to the terminating pod IP. Design a zero-downtime rolling deployment and graceful shutdown architecture.

From patterns to decisions.

Every time a high-throughput API gateway / payment service (20,000 RPS across 40 Kubernetes pods behind an AWS ALB and Envoy ingress) runs a rolling deployment, clients observe a spike of 502 Bad Gateway errors and TCP RSTs for 15–30 seconds. Investigation shows pods immediately close Netty/Tomcat listener sockets upon receiving SIGTERM while kube-proxy and Envoy still route new and keep-alive requests to the terminating pod IP. Design a zero-downtime rolling deployment and graceful shutdown architecture.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

Implementation & Guardrails (1).

Kubernetes triggers EndpointSlice removal and container SIGTERM concurrently. Because the JVM closes port 8080 in ~15 ms while Envoy/kube-proxy take 2-8s to update routing tables, thousands of requests hit a closed socket.

ChoiceWhat it buysWhat it costs
Trade-off 1Relying only on readinessProbe failure does not help once SIGTERM is sent, because kubelet doesn't wait for readinessProbe during deletion.
Trade-off 2Shortening readinessProbe periodSeconds increases API server load without fixing the t=0 race.

Trace t=0

Pod enters Terminating state.

Step 2

Compare control-plane propagation (API server -> EndpointSlice -> kube-proxy/Envoy/ALB = 2-8s) vs. JVM socket close (15 ms).

Step 3

Identify that HTTP/1.1 keep-alive connections already established by Envoy also fail if closed abruptly.

Alternative 1

Use AWS ALB TargetGroup IP mode with Pod Readiness Gates + preStop.

Alternative 2

Use Envoy graceful shutdown via Istio sidecar terminationDrainDuration.

Failure Mode 1

17,500+ dropped requests per deployment wave at 20,000 RPS.

Failure Mode 2

TCP RSTs on reused HTTP/1.1 keep-alive connections.

# Broken default configuration causing 502s
spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: pricing-api
      # Missing lifecycle.preStop -> SIGTERM arrives immediately at t=0
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Pair maxUnavailable.

Pair maxUnavailable: 0, preStop sleep 10, and terminationGracePeriodSeconds: 45 with Spring Boot server.shutdown=graceful and 65s keep-alive alignment.

ChoiceWhat it buysWhat it costs
maxSurge25% requires 10 extra pods worth of cluster capacity during rollouts.
Trade-off 210s preStop adds ~40s to a 4-wave deployment.

Step 1

Set strategy.rollingUpdate.maxUnavailable=0 and maxSurge=25%.

Step 2

Add preStop sleep 10 and terminationGracePeriodSeconds: 45.

Step 3

Enable server.shutdown=graceful and set timeout-per-shutdown-phase=30s.

Alternative 1

Use Blue/Green deployment with ALB weighted target groups for instant rollback.

Alternative 2

Use Argo Rollouts for automated canary analysis before scaling down old pods.

Failure Mode 1

If terminationGracePeriodSeconds is left at 30s while preStop is 10s, graceful drain only gets 20s before SIGKILL.

Failure Mode 2

Shell-form Docker ENTRYPOINT swallows SIGTERM, causing a 45s hang followed by SIGKILL.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-gateway
spec:
  replicas: 40
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 25%
  template:
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: payment-gateway
          image: registry.internal/payment-gateway:2.4.1
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 10"]
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            periodSeconds: 5
            failureThreshold: 2
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

If the server closes an.

If the server closes an idle HTTP/1.1 connection before the ALB/Envoy does, the proxy may dispatch a request into a closing socket causing a 502. Setting server idle timeout to 65s (>60s ALB) ensures the proxy closes first.

ChoiceWhat it buysWhat it costs
Trade-off 1Longer server keep-alive (65s) holds slightly more file descriptors per pod.
Trade-off 2HTTP/2 multiplexing avoids keep-alive races via GOAWAY but requires TLS/ALPN or h2c between Envoy and pod.

Step 1

Check upstream ALB/Envoy idle_timeout (typically 60s).

Step 2

Configure Spring Boot Tomcat/Netty keepAliveTimeout to 65s.

Step 3

Verify HTTP/2 server sends GOAWAY on SIGTERM so Envoy opens new streams on healthy pods.

Alternative 1

Disable keep-alive between Envoy and pods (adds TCP handshake latency).

Alternative 2

Configure Envoy max_connection_duration with jitter.

Failure Mode 1

Reverse inequality (server 55s < ALB 60s) produces intermittent 502s even outside deployments.

Failure Mode 2

Ignoring Envoy circuit-breaker max_connections on surge pods.

# application.yaml (Spring Boot 3)
server:
  shutdown: graceful
  tomcat:
    keep-alive-timeout: 65s
    max-keep-alive-requests: 10000
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s
management:
  endpoint:
    health:
      probes:
        enabled: true
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

Long-lived WebSocket/SSE streams will be.

Long-lived WebSocket/SSE streams will be force-closed at 45s unless the application handles SIGTERM by sending a reconnect/close frame with jittered backoff.

ChoiceWhat it buysWhat it costs
Trade-off 1Active WebSocket close triggers immediate client reconnects, requiring jitter to avoid a thundering herd on new pods.
Trade-off 2Extending terminationGracePeriodSeconds to 3600s for WebSockets blocks CI/CD rollouts for an hour.

Step 1

Listen for ContextClosedEvent in Java to notify active WebSocket/SSE clients.

Step 2

Send WebSocket CloseStatus.SERVICE_RESTARTED (1012) so clients reconnect with 1-5s jitter.

Step 3

Cap non-streaming HTTP requests with a 25s server timeout.

Alternative 1

Separate stateless REST pods from stateful WebSocket gateway pods.

Alternative 2

Use NATS/Redis Streams to replay missed messages after client reconnect.

Failure Mode 1

Reconnection storm if all 50,000 WebSocket clients reconnect simultaneously without jitter.

Failure Mode 2

Kubelet SIGKILL leaving half-open TCP sockets on clients without TCP keepalive.

@Component
public class GracefulWebSocketDrainer implements ApplicationListener<ContextClosedEvent> {
    private final SessionRegistry sessionRegistry;

    public GracefulWebSocketDrainer(SessionRegistry sessionRegistry) {
        this.sessionRegistry = sessionRegistry;
    }

    @Override
    public void onApplicationEvent(ContextClosedEvent event) {
        // Instruct clients to reconnect to healthy surge pods with jitter
        sessionRegistry.activeSessions().forEach(session -> 
            session.closeQuietly(CloseStatus.SERVICE_RESTARTED));
    }
}
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Pair the Deployment with a.

Pair the Deployment with a PodDisruptionBudget (minAvailable: 85%) so voluntary node drains respect the same zero-downtime capacity floor as rolling updates.

ChoiceWhat it buysWhat it costs
Setting minAvailable100% blocks all node upgrades and drains indefinitely.
Trade-off 285% allows 6 pods to drain concurrently while preserving enough capacity.

Step 1

Define a PodDisruptionBudget with minAvailable: 85% (or maxUnavailable: 15%).

Step 2

Ensure Kubernetes Cluster Autoscaler / Karpenter respects PDBs during consolidation.

Step 3

Verify preStop + graceful shutdown also runs on kubelet eviction/drain.

Alternative 1

Use maxUnavailable: 1 for ultra-critical small deployments (<10 replicas).

Alternative 2

Schedule node drains during off-peak hours via Karpenter Disruption Budgets.

Failure Mode 1

PDB deadlock if replicas=1 and minAvailable=1.

Failure Mode 2

Spot instance 2-minute interruption notice bypassing normal PDB if node-termination-handler is missing.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-gateway-pdb
spec:
  minAvailable: 85%
  selector:
    matchLabels:
      app: payment-gateway

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Kubernetes sends SIGTERM to the container in parallel with removing the pod from EndpointSlices; without a preStop delay, the app closes its listening socket before load balancers stop routing traffic to it.
  • 02Trade-off tip: Longer server keep-alive (65s) holds slightly more file descriptors per pod.
  • 03Trade-off tip: HTTP/2 multiplexing avoids keep-alive races via GOAWAY but requires TLS/ALPN or h2c between Envoy and pod.
  • 04Watch out for: Reconnection storm if all 50,000 WebSocket clients reconnect simultaneously without jitter.
  • 05Watch out for: Kubelet SIGKILL leaving half-open TCP sockets on clients without TCP keepalive.
Behavioral Progression · Graceful Shutdown Rolling Deployments

Graceful Shutdown Rolling Deployments — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Kubernetes sends SIGTERM to the container in parallel with removing the pod from EndpointSlices; without a preStop delay, the app closes its listening socket before load balancers stop routing traffic to it.

Synthesis / One connected design

Putting Graceful Shutdown in Rolling Deployments Together

Kubernetes sends SIGTERM to the container in parallel with removing the pod from EndpointSlices; without a preStop delay, the app closes its listening socket before load balancers stop routing traffic to it.

A candidate architecture, not a universal answer

Putting Graceful Shutdown in Rolling Deployments Together

Step 1
Step 2
Step 3
  1. Set strategy.rollingUpdate.maxUnavailable=0 and maxSurge=25%.
  2. Add preStop sleep 10 and terminationGracePeriodSeconds: 45.
  3. Enable server.shutdown=graceful and set timeout-per-shutdown-phase=30s.
Long-lived WebSocket/SSE streams will be force-closed at 45s unless the application handles SIGTERM by sending a reconnect/close frame with jittered backoff.
Interview Readiness Check

Long-lived WebSocket/SSE streams will be force-closed at 45s unless the application handles SIGTERM by sending a reconnect/close frame with jittered backoff.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Graceful Shutdown in Rolling Deployments

Kubernetes sends SIGTERM to the container in parallel with removing the pod from EndpointSlices; without a preStop delay, the app closes its listening socket before load balancers stop routing traffic to it.