A Java Spring Boot order-processing service (24 pods on Kubernetes, HikariCP DB pool = 20, Tomcat max threads = 200) experiences a brief 4-second downstream Postgres lock contention. Because `/actuator/health` is wired to both `livenessProbe` and `readinessProbe` (`timeoutSeconds: 1`, `periodSeconds: 5`, `failureThreshold: 3`) and shares Tomcat's worker pool, all 24 pods fail liveness checks after 15 seconds. Kubelet kills and restarts all 24 pods simultaneously, causing a 6-minute outage due to 45-second JVM cold starts and cold-cache DB stampedes. Redesign the health-check and overload isolation architecture.
From patterns to decisions.
A Java Spring Boot order-processing service (24 pods on Kubernetes, HikariCP DB pool = 20, Tomcat max threads = 200) experiences a brief 4-second downstream Postgres lock contention. Because `/actuator/health` is wired to both `livenessProbe` and `readinessProbe` (`timeoutSeconds: 1`, `periodSeconds: 5`, `failureThreshold: 3`) and shares Tomcat's worker pool, all 24 pods fail liveness checks after 15 seconds. Kubelet kills and restarts all 24 pods simultaneously, causing a 6-minute outage due to 45-second JVM cold starts and cold-cache DB stampedes. Redesign the health-check and overload isolation architecture.
Liveness checked the DB and.
Liveness checked the DB and shared Tomcat's blocked worker pool with a 1s timeout; all 24 pods failed 3 checks in 15s and entered 45s cold-start loops.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Checking DB in liveness can never fix a remote Postgres lock; it only destroys healthy app pods. |
Trace the 2-way failure
DB check fails + Tomcat thread queue blocks /actuator/health.
Step 2
Show how 24 simultaneous 45s restarts + cold-cache DB stampede turns 4s into 6 minutes.
Alternative 1
Use readinessProbe to remove overloaded pods from Service endpoints.
Failure Mode 1
All pods restarting simultaneously saturates CPU and DB connection handshakes.
# Anti-pattern that caused the outage:
livenessProbe:
httpGet:
path: /actuator/health
port: 8080
timeoutSeconds: 1
periodSeconds: 5
failureThreshold: 3
Separate management onto port 8081.
Separate management onto port 8081, bind liveness strictly to livenessState, readiness to readinessState,db, and configure startupProbe.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Port 8081 ensures `/livez` responds even when port 8080's 200-thread pool is 100% blocked. |
Set `management.server.port
8081`.
Configure `group.liveness.include
livenessState and group.readiness.include: readinessState,db`.
Step 3
Add startupProbe, readinessProbe, and livenessProbe in K8s spec.
Alternative 1
Use gRPC health probe or a dedicated lightweight Netty HTTP server for /livez.
Failure Mode 1
Exposing port 8081 on the public Ingress/Service instead of keeping it cluster-internal.
# application.yaml
management:
server:
port: 8081
endpoint:
health:
probes:
enabled: true
group:
liveness:
include: livenessState
readiness:
include: readinessState,db
spring:
datasource:
hikari:
connection-timeout: 1500
---
# Kubernetes Pod Spec
startupProbe:
httpGet: { path: /actuator/health/liveness, port: 8081 }
periodSeconds: 5
failureThreshold: 18
readinessProbe:
httpGet: { path: /actuator/health/readiness, port: 8081 }
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2
successThreshold: 2
livenessProbe:
httpGet: { path: /actuator/health/liveness, port: 8081 }
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 6
Yes, for single-pod DB network.
Yes, for single-pod DB network splits, readiness should fail; however, if Postgres itself is down, all 24 pods failing readiness causes kube-proxy to drop all endpoints (503/connection refused) unless publishNotReadyAddresses or fast-fail 503s are preferred.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Fast 503s keep pods in the Service pool so recovery is instantaneous when the DB lock clears. |
Step 1
Distinguish local pod DB-pool exhaustion from global DB outage.
Step 2
Explain that returning HTTP 503 with Retry-After from warm pods is often cleaner than removing all 24 endpoints.
Alternative 1
Use Resilience4j circuit breaker to trip open after 50% DB failures.
Failure Mode 1
Removing all 24 endpoints causes kube-proxy to send TCP RST (connection refused) at the Ingress.
// Fail fast with 503 via Hikari timeout instead of blocking threads
@RestControllerAdvice
public class PoolOverloadHandler {
@ExceptionHandler(SQLTransientConnectionException.class)
public ResponseEntity<String> handlePoolExhausted() {
return ResponseEntity.status(503).header("Retry-After", "2").body("DB pool saturated");
}
}
Linux CFS CPU quota throttling.
Linux CFS CPU quota throttling (limits.cpu: 2) freezes the entire JVM process—including port 8081 threads—for up to 90+ ms per 100 ms period.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Omitting `limits.cpu` eliminates CFS throttling pauses while `requests.cpu` still guarantees fair share under contention. |
Step 1
Check container_cpu_cfs_throttled_periods_total.
Step 2
Remove limits.cpu (keep requests.cpu) or raise timeoutSeconds: 5 and limits.cpu headroom.
Alternative 1
Use timeoutSeconds: 5 + failureThreshold: 6 so transient 200 ms CFS pauses never accumulate 6 consecutive failures.
Failure Mode 1
Strict limits.cpu with multi-threaded JVM GC causes severe tail-latency and probe timeouts.
resources:
requests:
cpu: "2"
memory: "4Gi"
limits:
memory: "4Gi"
# Omit limits.cpu or set higher burst headroom to avoid CFS throttling
Only unrecoverable internal failures.
Only unrecoverable internal failures: OutOfMemoryError (-XX:+ExitOnOutOfMemoryError exits directly without needing liveness), or an internal deadlock detected by ThreadMXBean.findDeadlockedThreads().
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | `findDeadlockedThreads()` takes ~1-2 ms and inspects JVM monitor/ownable synchronizer cycles. |
Use `-XX
+ExitOnOutOfMemoryError` for immediate OOM exit.
Step 2
Optionally check ThreadMXBean.findDeadlockedThreads() in a custom liveness indicator.
Alternative 1
Keep liveness strictly as a lightweight 200 OK ping and rely on -XX:+ExitOnOutOfMemoryError.
Failure Mode 1
Putting complex lock checks inside liveness without caching can itself add CPU overhead.
import java.lang.management.ManagementFactory;
import java.lang.management.ThreadMXBean;
public boolean isJvmDeadlocked() {
ThreadMXBean bean = ManagementFactory.getThreadMXBean();
long[] deadlocked = bean.findDeadlockedThreads();
return deadlocked != null && deadlocked.length > 0;
}
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Separate
/livez(dependency-free, dedicated lightweight thread/listener) from/readyz(dependency & pool capacity checks), addstartupProbe, and size probe timeouts above GC/CFS pauses. - 02Trade-off tip: Fast 503s keep pods in the Service pool so recovery is instantaneous when the DB lock clears.
- 03Watch out for: Strict
limits.cpuwith multi-threaded JVM GC causes severe tail-latency and probe timeouts.
Designing Safe Liveness Readiness Probes — Normal → Load / Fault → Recovery
Separate /livez (dependency-free, dedicated lightweight thread/listener) from /readyz (dependency & pool capacity checks), add startupProbe, and size probe timeouts above GC/CFS pauses.
Putting Designing Safe Liveness & Readiness Probes Together
Separate `/livez` (dependency-free, dedicated lightweight thread/listener) from `/readyz` (dependency & pool capacity checks), add `startupProbe`, and size probe timeouts above GC/CFS pauses.
Putting Designing Safe Liveness & Readiness Probes Together
- Set
management.server.port: 8081. - Configure
group.liveness.include: livenessStateandgroup.readiness.include: readinessState,db. - Add
startupProbe,readinessProbe, andlivenessProbein K8s spec.
Interview Readiness Check
Linux CFS CPU quota throttling (limits.cpu: 2) freezes the entire JVM process—including port 8081 threads—for up to 90+ ms per 100 ms period.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Designing Safe Liveness & Readiness Probes
Separate /livez (dependency-free, dedicated lightweight thread/listener) from /readyz (dependency & pool capacity checks), add startupProbe, and size probe timeouts above GC/CFS pauses.
