Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Fixing Stale DNS Caching
in Long-Lived Clients.

Your payment orchestration service (Java 17 + Apache HttpClient 5 / Netty WebClient) communicates with an external partner's AWS Application Load Balancer (api.partner-pay.com, DNS TTL = 60s). When AWS scales or replaces the ALB nodes, the DNS A-records rotate to new IPs and the old ALB IPs are decommissioned 5 minutes later. However, your long-lived pods continue sending 100% of traffic to the dead IP for hours—triggering ConnectTimeoutException and 504 Gateway Timeout until pods are manually restarted. Fix both the DNS cache layer and the persistent HTTP connection pool.
Start the journey5 stages
Loading diagram…
Fixing Stale DNS Caching in Long-Lived Clients architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your payment orchestration service (Java 17 + Apache HttpClient 5 / Netty WebClient) communicates with an external partner's AWS Application Load Balancer (`api.partner-pay.com`, DNS TTL = 60s). When AWS scales or replaces the ALB nodes, the DNS A-records rotate to new IPs and the old ALB IPs are decommissioned 5 minutes later. However, your long-lived pods continue sending 100% of traffic to the dead IP for hours—triggering `ConnectTimeoutException` and `504 Gateway Timeout` until pods are manually restarted. Fix both the DNS cache layer and the persistent HTTP connection pool.

From patterns to decisions.

Your payment orchestration service (Java 17 + Apache HttpClient 5 / Netty WebClient) communicates with an external partner's AWS Application Load Balancer (`api.partner-pay.com`, DNS TTL = 60s). When AWS scales or replaces the ALB nodes, the DNS A-records rotate to new IPs and the old ALB IPs are decommissioned 5 minutes later. However, your long-lived pods continue sending 100% of traffic to the dead IP for hours—triggering `ConnectTimeoutException` and `504 Gateway Timeout` until pods are manually restarted. Fix both the DNS cache layer and the persistent HTTP connection pool.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

Implementation & Guardrails (1).

Keep-alive TCP/TLS connections only perform DNS resolution at socket creation; an already-open socket continues sending bytes to the old IP until closed.

ChoiceWhat it buysWhat it costs
Keeping `Connectionclose` solves pinning but destroys TLS 1.3 performance.

Step 1

Check java.security / JVM flags for networkaddress.cache.ttl.

Step 2

Inspect HTTP client pool timeToLive / maxLifeTime settings.

Alternative 1

Sidecar Envoy proxy with strict_dns cluster and dns_refresh_rate: 30s.

Failure Mode 1

Setting Security.setProperty after Spring Boot initializes InetAddress has no effect.

JAVA_TOOL_OPTIONS: "-Dnetworkaddress.cache.ttl=30 -Dnetworkaddress.cache.negative.ttl=1"
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Combine networkaddress.

Combine networkaddress.cache.ttl=30 with setTimeToLive(60s) and connectTimeout(2s) so sockets recycle gracefully and re-resolve DNS.

ChoiceWhat it buysWhat it costs
Trade-off 160s `timeToLive` costs ~1.6 TLS handshakes/sec per pod while guaranteeing < 60s ALB IP convergence.

Step 1

Set JVM networkaddress.cache.ttl=30 and negative.ttl=1 via JAVA_TOOL_OPTIONS.

Step 2

Set timeToLive=60s and maxIdleTime=20s on PoolingHttpClientConnectionManager.

Step 3

Set connectTimeout=2s so dead IPs fail fast instead of hanging for 30s.

Alternative 1

Python urllib3/httpx pool eviction via periodic poolmanager.clear() or Envoy sidecar.

Failure Mode 1

Without evictInBackground, idle connections sit open until the next request hits a dead socket.

ConnectionConfig.custom()
    .setConnectTimeout(Timeout.ofSeconds(2))
    .setTimeToLive(TimeValue.ofSeconds(60))
    .setValidateAfterInactivity(TimeValue.ofSeconds(5))
    .build();
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

HTTP/2 multiplexes all concurrent requests.

HTTP/2 multiplexes all concurrent requests onto a single TCP connection, so one stale socket pins 100% of a pod's traffic to a single backend IP.

ChoiceWhat it buysWhat it costs
Trade-off 1Envoy `strict_dns` + `max_connection_duration` handles rotation uniformly across Java, Python, and Go services.

Step 1

Verify protocol negotiation (ALPN h2 vs http/1.1).

Step 2

Enforce GOAWAY / connection rotation via maxLifeTime or sidecar Envoy.

Alternative 1

gRPC client-side round_robin load balancing with DNS xds or dns:///.

Failure Mode 1

Closing an active HTTP/2 socket abruptly causes StreamResetException; must use graceful GOAWAY.

# Envoy cluster config for HTTP/2 upstream
common_http_protocol_options:
  max_connection_duration: 60s
dns_refresh_rate: 30s
lb_policy: ROUND_ROBIN
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

Deploy Kubernetes NodeLocal DNSCache and.

Deploy Kubernetes NodeLocal DNSCache and jitter connection lifetimes so DNS lookups are served locally from node memory over TCP/UDP.

ChoiceWhat it buysWhat it costs
Trade-off 1`serve_stale` prevents outages during brief CoreDNS hiccups at the cost of serving slightly older records during DNS failures.

Step 1

Enable NodeLocal DNSCache DaemonSet (169.254.20.10) to bypass kube-proxy conntrack.

Step 2

Enable CoreDNS cache 30 with serve_stale.

Alternative 1

Staggering pod startup and jittering maxLifeTime (50s-70s).

Failure Mode 1

Linux conntrack table exhaustion on high-QPS UDP DNS queries without NodeLocal DNSCache.

.:53 {
    cache 30 {
        serve_stale 5m
    }
    forward . /etc/resolv.conf
}
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Run a chaos test that.

Run a chaos test that updates a mock DNS A-record to IP2, waits 60s, then drops packets to IP1 via iptables while asserting 0% 5xx rate.

ChoiceWhat it buysWhat it costs
Trade-off 1Automated chaos DNS rotation tests catch regressions when libraries or base images are upgraded.

Step 1

Point service at a controllable CoreDNS zone with TTL=15s.

Step 2

Swap A-record from IP1 to IP2, wait 60s, and blackhole IP1.

Alternative 1

Monitoring active connection age histograms (http_client_connection_age_seconds).

Testing only with `Connection

close` curl instead of long-lived load-tested pods.

# Synthetic chaos assertion
assert error_rate_during_ip_drain < 0.001, f'Stale IP pinning detected: {error_rate_during_ip_drain}'

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Align DNS resolution TTL (networkaddress.cache.ttl=30), cap HTTP connection maxConnectionLifeTime (60s), and enable proactive DNS refresh in Netty/HttpClient.
  • 02Trade-off tip: Envoy strict_dns + max_connection_duration handles rotation uniformly across Java, Python, and Go services.
  • 03Watch out for: Linux conntrack table exhaustion on high-QPS UDP DNS queries without NodeLocal DNSCache.
Behavioral Progression · Fixing Stale Dns Caching Clients

Fixing Stale Dns Caching Clients — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Align DNS resolution TTL (networkaddress.cache.ttl=30), cap HTTP connection maxConnectionLifeTime (60s), and enable proactive DNS refresh in Netty/HttpClient.

Synthesis / One connected design

Putting Fixing Stale DNS Caching in Long-Lived Clients Together

Align DNS resolution TTL (`networkaddress.cache.ttl=30`), cap HTTP connection `maxConnectionLifeTime` (60s), and enable proactive DNS refresh in `Netty`/`HttpClient`.

A candidate architecture, not a universal answer

Putting Fixing Stale DNS Caching in Long-Lived Clients Together

Step 1
Step 2
Step 3
  1. Set JVM networkaddress.cache.ttl=30 and negative.ttl=1 via JAVA_TOOL_OPTIONS.
  2. Set timeToLive=60s and maxIdleTime=20s on PoolingHttpClientConnectionManager.
  3. Set connectTimeout=2s so dead IPs fail fast instead of hanging for 30s.
Deploy Kubernetes `NodeLocal DNSCache` and jitter connection lifetimes so DNS lookups are served locally from node memory over TCP/UDP.
Interview Readiness Check

Deploy Kubernetes NodeLocal DNSCache and jitter connection lifetimes so DNS lookups are served locally from node memory over TCP/UDP.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Fixing Stale DNS Caching in Long-Lived Clients

Align DNS resolution TTL (networkaddress.cache.ttl=30), cap HTTP connection maxConnectionLifeTime (60s), and enable proactive DNS refresh in Netty/HttpClient.