Your payment orchestration service (Java 17 + Apache HttpClient 5 / Netty WebClient) communicates with an external partner's AWS Application Load Balancer (`api.partner-pay.com`, DNS TTL = 60s). When AWS scales or replaces the ALB nodes, the DNS A-records rotate to new IPs and the old ALB IPs are decommissioned 5 minutes later. However, your long-lived pods continue sending 100% of traffic to the dead IP for hours—triggering `ConnectTimeoutException` and `504 Gateway Timeout` until pods are manually restarted. Fix both the DNS cache layer and the persistent HTTP connection pool.
From patterns to decisions.
Your payment orchestration service (Java 17 + Apache HttpClient 5 / Netty WebClient) communicates with an external partner's AWS Application Load Balancer (`api.partner-pay.com`, DNS TTL = 60s). When AWS scales or replaces the ALB nodes, the DNS A-records rotate to new IPs and the old ALB IPs are decommissioned 5 minutes later. However, your long-lived pods continue sending 100% of traffic to the dead IP for hours—triggering `ConnectTimeoutException` and `504 Gateway Timeout` until pods are manually restarted. Fix both the DNS cache layer and the persistent HTTP connection pool.
Implementation & Guardrails (1).
Keep-alive TCP/TLS connections only perform DNS resolution at socket creation; an already-open socket continues sending bytes to the old IP until closed.
| Choice | What it buys | What it costs |
|---|---|---|
| Keeping `Connection | close` solves pinning but destroys TLS 1.3 performance. |
Step 1
Check java.security / JVM flags for networkaddress.cache.ttl.
Step 2
Inspect HTTP client pool timeToLive / maxLifeTime settings.
Alternative 1
Sidecar Envoy proxy with strict_dns cluster and dns_refresh_rate: 30s.
Failure Mode 1
Setting Security.setProperty after Spring Boot initializes InetAddress has no effect.
JAVA_TOOL_OPTIONS: "-Dnetworkaddress.cache.ttl=30 -Dnetworkaddress.cache.negative.ttl=1"
Combine networkaddress.
Combine networkaddress.cache.ttl=30 with setTimeToLive(60s) and connectTimeout(2s) so sockets recycle gracefully and re-resolve DNS.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | 60s `timeToLive` costs ~1.6 TLS handshakes/sec per pod while guaranteeing < 60s ALB IP convergence. |
Step 1
Set JVM networkaddress.cache.ttl=30 and negative.ttl=1 via JAVA_TOOL_OPTIONS.
Step 2
Set timeToLive=60s and maxIdleTime=20s on PoolingHttpClientConnectionManager.
Step 3
Set connectTimeout=2s so dead IPs fail fast instead of hanging for 30s.
Alternative 1
Python urllib3/httpx pool eviction via periodic poolmanager.clear() or Envoy sidecar.
Failure Mode 1
Without evictInBackground, idle connections sit open until the next request hits a dead socket.
ConnectionConfig.custom()
.setConnectTimeout(Timeout.ofSeconds(2))
.setTimeToLive(TimeValue.ofSeconds(60))
.setValidateAfterInactivity(TimeValue.ofSeconds(5))
.build();
HTTP/2 multiplexes all concurrent requests.
HTTP/2 multiplexes all concurrent requests onto a single TCP connection, so one stale socket pins 100% of a pod's traffic to a single backend IP.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Envoy `strict_dns` + `max_connection_duration` handles rotation uniformly across Java, Python, and Go services. |
Step 1
Verify protocol negotiation (ALPN h2 vs http/1.1).
Step 2
Enforce GOAWAY / connection rotation via maxLifeTime or sidecar Envoy.
Alternative 1
gRPC client-side round_robin load balancing with DNS xds or dns:///.
Failure Mode 1
Closing an active HTTP/2 socket abruptly causes StreamResetException; must use graceful GOAWAY.
# Envoy cluster config for HTTP/2 upstream
common_http_protocol_options:
max_connection_duration: 60s
dns_refresh_rate: 30s
lb_policy: ROUND_ROBIN
Deploy Kubernetes NodeLocal DNSCache and.
Deploy Kubernetes NodeLocal DNSCache and jitter connection lifetimes so DNS lookups are served locally from node memory over TCP/UDP.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | `serve_stale` prevents outages during brief CoreDNS hiccups at the cost of serving slightly older records during DNS failures. |
Step 1
Enable NodeLocal DNSCache DaemonSet (169.254.20.10) to bypass kube-proxy conntrack.
Step 2
Enable CoreDNS cache 30 with serve_stale.
Alternative 1
Staggering pod startup and jittering maxLifeTime (50s-70s).
Failure Mode 1
Linux conntrack table exhaustion on high-QPS UDP DNS queries without NodeLocal DNSCache.
.:53 {
cache 30 {
serve_stale 5m
}
forward . /etc/resolv.conf
}
Run a chaos test that.
Run a chaos test that updates a mock DNS A-record to IP2, waits 60s, then drops packets to IP1 via iptables while asserting 0% 5xx rate.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Automated chaos DNS rotation tests catch regressions when libraries or base images are upgraded. |
Step 1
Point service at a controllable CoreDNS zone with TTL=15s.
Step 2
Swap A-record from IP1 to IP2, wait 60s, and blackhole IP1.
Alternative 1
Monitoring active connection age histograms (http_client_connection_age_seconds).
Testing only with `Connection
close` curl instead of long-lived load-tested pods.
# Synthetic chaos assertion
assert error_rate_during_ip_drain < 0.001, f'Stale IP pinning detected: {error_rate_during_ip_drain}'
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Align DNS resolution TTL (
networkaddress.cache.ttl=30), cap HTTP connectionmaxConnectionLifeTime(60s), and enable proactive DNS refresh inNetty/HttpClient. - 02Trade-off tip: Envoy
strict_dns+max_connection_durationhandles rotation uniformly across Java, Python, and Go services. - 03Watch out for: Linux
conntracktable exhaustion on high-QPS UDP DNS queries without NodeLocal DNSCache.
Fixing Stale Dns Caching Clients — Normal → Load / Fault → Recovery
Align DNS resolution TTL (networkaddress.cache.ttl=30), cap HTTP connection maxConnectionLifeTime (60s), and enable proactive DNS refresh in Netty/HttpClient.
Putting Fixing Stale DNS Caching in Long-Lived Clients Together
Align DNS resolution TTL (`networkaddress.cache.ttl=30`), cap HTTP connection `maxConnectionLifeTime` (60s), and enable proactive DNS refresh in `Netty`/`HttpClient`.
Putting Fixing Stale DNS Caching in Long-Lived Clients Together
- Set JVM
networkaddress.cache.ttl=30andnegative.ttl=1viaJAVA_TOOL_OPTIONS. - Set
timeToLive=60sandmaxIdleTime=20sonPoolingHttpClientConnectionManager. - Set
connectTimeout=2sso dead IPs fail fast instead of hanging for 30s.
Interview Readiness Check
Deploy Kubernetes NodeLocal DNSCache and jitter connection lifetimes so DNS lookups are served locally from node memory over TCP/UDP.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Fixing Stale DNS Caching in Long-Lived Clients
Align DNS resolution TTL (networkaddress.cache.ttl=30), cap HTTP connection maxConnectionLifeTime (60s), and enable proactive DNS refresh in Netty/HttpClient.
