Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Aligning HTTP Keep-Alive
Timeouts Across Proxies.

Your checkout web app sits behind an AWS Application Load Balancer (default idle_timeout = 60s) and an Nginx ingress (keepalive_timeout = 65s), forwarding to Spring Boot Tomcat (server.tomcat.keep-alive-timeout = 60s) and a Node.js microservice (server.keepAliveTimeout = 5s). Under steady traffic, 0.3% of POST /orders requests fail randomly with 502 Bad Gateway and upstream prematurely closed connection / ECONNRESET in Nginx/ALB logs—even though backend pods show 0 errors and <20% CPU. Diagnose the TCP race condition and fix the timeout hierarchy.
Start the journey5 stages
Loading diagram…
Aligning HTTP Keep-Alive Timeouts Across Proxies architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your checkout web app sits behind an AWS Application Load Balancer (default `idle_timeout = 60s`) and an Nginx ingress (`keepalive_timeout = 65s`), forwarding to Spring Boot Tomcat (`server.tomcat.keep-alive-timeout = 60s`) and a Node.js microservice (`server.keepAliveTimeout = 5s`). Under steady traffic, 0.3% of `POST /orders` requests fail randomly with `502 Bad Gateway` and `upstream prematurely closed connection` / `ECONNRESET` in Nginx/ALB logs—even though backend pods show 0 errors and <20% CPU. Diagnose the TCP race condition and fix the timeout hierarchy.

TL;DR30-second simplified summary
▾

Your checkout web app sits behind an AWS Application Load Balancer (default idle_timeout = 60s) and an Nginx ingress (keepalive_timeout = 65s), forwarding to Spring Boot Tomcat (server.tomcat.keep-alive-timeout = 60s) and a Node.js microservice (server.keepAliveTimeout = 5s). Under steady traffic, 0.3% of POST /orders requests fail randomly with 502 Bad Gateway and upstream prematurely closed connection / ECONNRESET in Nginx/ALB logs—even though backend pods show 0 errors and <20% CPU. Diagnose the TCP race condition and fix the timeout hierarchy.

Always set the backend's keepAliveTimeout strictly greater than the upstream proxy's idle timeout (e.g., ALB 60s -> Backend 75s), and let the proxy initiate idle closes.

From patterns to decisions.

Your checkout web app sits behind an AWS Application Load Balancer (default `idle_timeout = 60s`) and an Nginx ingress (`keepalive_timeout = 65s`), forwarding to Spring Boot Tomcat (`server.tomcat.keep-alive-timeout = 60s`) and a Node.js microservice (`server.keepAliveTimeout = 5s`). Under steady traffic, 0.3% of `POST /orders` requests fail randomly with `502 Bad Gateway` and `upstream prematurely closed connection` / `ECONNRESET` in Nginx/ALB logs—even though backend pods show 0 errors and <20% CPU. Diagnose the TCP race condition and fix the timeout hierarchy.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

The backend closes an idle.

The backend closes an idle TCP socket (FIN) right as the proxy writes a new request to it. The backend kernel replies with TCP RST before the HTTP layer ever sees the request.

ChoiceWhat it buysWhat it costs
Trade-off 1Keep-alive eliminates TCP handshake latency, but requires asymmetric timeout ownership.

Check default timeouts

Node.js (5s), Tomcat (60s), ALB (60s), Nginx (65s).

Step 2

Observe that any backend timeout <= upstream proxy timeout creates a race window.

Alternative 1

Use HTTP/2 upstream where GOAWAY prevents write-after-close races.

Failure Mode 1

Backend logs remain completely clean, misleading engineers into blaming the network.

# Classic 502 race timeline:
# t=5.000s: Node.js sends TCP FIN
# t=5.0001s: Nginx writes POST /orders to the same socket
# t=5.0002s: Node.js kernel responds TCP RST -> Nginx returns 502
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Set Nginx server keepalive_timeout to.

Set Nginx server keepalive_timeout to 75s (> ALB 60s), Nginx upstream keepalive_timeout to 55s, and both Tomcat and Node.js keepAliveTimeout to 90s.

ChoiceWhat it buysWhat it costs
Trade-off 190s backend timeout holds idle sockets slightly longer, using negligible RAM while eliminating 502s.

Step 1

Configure Nginx server block (75s) and upstream block (55s).

Step 2

Configure Spring Boot server.tomcat.keep-alive-timeout: 90s.

Step 3

Configure Node.js server.keepAliveTimeout = 90000 and server.headersTimeout = 95000.

Alternative 1

In Nginx, set upstream { keepalive_timeout 55s; keepalive_requests 5000; }.

Failure Mode 1

Forgetting headersTimeout > keepAliveTimeout in Node.js still triggers premature closes.

# Spring Boot application.yaml
server:
  tomcat:
    keep-alive-timeout: 90s
    max-keep-alive-requests: 10000

# Node.js equivalent:
# server.keepAliveTimeout = 90000;
# server.headersTimeout = 95000;
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

Retrying error on non-idempotent POST.

Retrying error on non-idempotent POST /orders is dangerous (and disabled by default in Nginx) because a socket error after partial write can duplicate orders.

ChoiceWhat it buysWhat it costs
Trade-off 1Fixing timeout ordering prevents the error at the source for all HTTP methods including `POST`.

Step 1

Explain that proxy_next_upstream skips POST unless non_idempotent is explicitly enabled.

Step 2

Even for GET, retrying masks the root cause and adds tail latency.

Alternative 1

Add Idempotency-Key support if POST retries are ever needed for real crashes.

Failure Mode 1

Enabling non_idempotent retries causes double-charges during slow backend responses.

# Nginx upstream block (safe)
upstream backend_pool {
  server app-service:8080;
  keepalive 128;
  keepalive_timeout 55s;
  keepalive_requests 5000;
}
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (4).

Tomcat's default maxKeepAliveRequests is 100. At 6,000 RPS across 30 pods (200 RPS/pod), each socket hits 100 requests every 0.5 seconds and closes!

ChoiceWhat it buysWhat it costs
Trade-off 1Higher `maxKeepAliveRequests` maximizes socket reuse; ensure load balancer still spreads traffic across new pods.

Step 1

Raise server.tomcat.max-keep-alive-requests from 100 to 10000 (or -1).

Step 2

Align Nginx keepalive_requests 5000 so Nginx closes the socket before Tomcat's 10000.

Alternative 1

Use maxKeepAliveRequests = -1 (unlimited) and let Nginx's keepalive_requests 5000 manage rotation.

Failure Mode 1

Leaving maxKeepAliveRequests = 100 recreates the exact same FIN/write race via request count instead of time.

server:
  tomcat:
    keep-alive-timeout: 90s
    max-keep-alive-requests: 10000 # Must exceed Nginx keepalive_requests (5000)
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Capture TCP FIN flags with.

Capture TCP FIN flags with tcpdump on the backend port (8080) and verify the source IP of FIN packets is Nginx, not the pod.

ChoiceWhat it buysWhat it costs
Trade-off 1Quick 30-second `tcpdump` validates the fix definitively.

Step 1

Run tcpdump -ni any 'tcp[tcpflags] & tcp-fin != 0 and port 8080'.

Step 2

Confirm upstream prematurely closed connection drops to 0 in Nginx metrics.

Alternative 1

Monitor nginx_ingress_controller_requests{status="502"} in Prometheus.

Failure Mode 1

Seeing RST from backend port 8080 indicates another timeout (e.g., requestTimeout or maxKeepAliveRequests) is still firing.

# Verify FIN direction on backend pod
# tcpdump -ni eth0 'tcp[tcpflags] & (tcp-fin|tcp-rst) != 0 and port 8080'

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Always set the backend's keepAliveTimeout strictly greater than the upstream proxy's idle timeout (e.g., ALB 60s -> Backend 75s), and let the proxy initiate idle closes.
  • 02Trade-off tip: Fixing timeout ordering prevents the error at the source for all HTTP methods including POST.
  • 03Watch out for: Leaving maxKeepAliveRequests = 100 recreates the exact same FIN/write race via request count instead of time.
Put it together · Synthesis

Putting Aligning HTTP Keep-Alive Timeouts Across Proxies Together

Always set the backend's `keepAliveTimeout` strictly greater than the upstream proxy's idle timeout (e.g., ALB 60s -> Backend 75s), and let the proxy initiate idle closes.

A candidate architecture, not a universal answer

Putting Aligning HTTP Keep-Alive Timeouts Across Proxies Together

Step 1
Step 2
Step 3
  1. Configure Nginx server block (75s) and upstream block (55s).
  2. Configure Spring Boot server.tomcat.keep-alive-timeout: 90s.
  3. Configure Node.js server.keepAliveTimeout = 90000 and server.headersTimeout = 95000.
Tomcat's default `maxKeepAliveRequests` is `100`. At 6,000 RPS across 30 pods (200 RPS/pod), each socket hits 100 requests every 0.5 seconds and closes!
Interview Readiness Check

Tomcat's default maxKeepAliveRequests is 100. At 6,000 RPS across 30 pods (200 RPS/pod), each socket hits 100 requests every 0.5 seconds and closes!

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Aligning HTTP Keep-Alive Timeouts Across Proxies

Always set the backend's keepAliveTimeout strictly greater than the upstream proxy's idle timeout (e.g., ALB 60s -> Backend 75s), and let the proxy initiate idle closes.