Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Designing Safe Liveness
& Readiness Probes.

A Java Spring Boot order-processing service (24 pods on Kubernetes, HikariCP DB pool = 20, Tomcat max threads = 200) experiences a brief 4-second downstream Postgres lock contention. Because /actuator/health is wired to both livenessProbe and readinessProbe (timeoutSeconds: 1, periodSeconds: 5, failureThreshold: 3) and shares Tomcat's worker pool, all 24 pods fail liveness checks after 15 seconds. Kubelet kills and restarts all 24 pods simultaneously, causing a 6-minute outage due to 45-second JVM cold starts and cold-cache DB stampedes. Redesign the health-check and overload isolation architecture.
Start the journey5 stages
Loading diagram…
Designing Safe Liveness & Readiness Probes architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A Java Spring Boot order-processing service (24 pods on Kubernetes, HikariCP DB pool = 20, Tomcat max threads = 200) experiences a brief 4-second downstream Postgres lock contention. Because `/actuator/health` is wired to both `livenessProbe` and `readinessProbe` (`timeoutSeconds: 1`, `periodSeconds: 5`, `failureThreshold: 3`) and shares Tomcat's worker pool, all 24 pods fail liveness checks after 15 seconds. Kubelet kills and restarts all 24 pods simultaneously, causing a 6-minute outage due to 45-second JVM cold starts and cold-cache DB stampedes. Redesign the health-check and overload isolation architecture.

From patterns to decisions.

A Java Spring Boot order-processing service (24 pods on Kubernetes, HikariCP DB pool = 20, Tomcat max threads = 200) experiences a brief 4-second downstream Postgres lock contention. Because `/actuator/health` is wired to both `livenessProbe` and `readinessProbe` (`timeoutSeconds: 1`, `periodSeconds: 5`, `failureThreshold: 3`) and shares Tomcat's worker pool, all 24 pods fail liveness checks after 15 seconds. Kubelet kills and restarts all 24 pods simultaneously, causing a 6-minute outage due to 45-second JVM cold starts and cold-cache DB stampedes. Redesign the health-check and overload isolation architecture.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

Liveness checked the DB and.

Liveness checked the DB and shared Tomcat's blocked worker pool with a 1s timeout; all 24 pods failed 3 checks in 15s and entered 45s cold-start loops.

ChoiceWhat it buysWhat it costs
Trade-off 1Checking DB in liveness can never fix a remote Postgres lock; it only destroys healthy app pods.

Trace the 2-way failure

DB check fails + Tomcat thread queue blocks /actuator/health.

Step 2

Show how 24 simultaneous 45s restarts + cold-cache DB stampede turns 4s into 6 minutes.

Alternative 1

Use readinessProbe to remove overloaded pods from Service endpoints.

Failure Mode 1

All pods restarting simultaneously saturates CPU and DB connection handshakes.

# Anti-pattern that caused the outage:
livenessProbe:
  httpGet:
    path: /actuator/health
    port: 8080
  timeoutSeconds: 1
  periodSeconds: 5
  failureThreshold: 3
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Separate management onto port 8081.

Separate management onto port 8081, bind liveness strictly to livenessState, readiness to readinessState,db, and configure startupProbe.

ChoiceWhat it buysWhat it costs
Trade-off 1Port 8081 ensures `/livez` responds even when port 8080's 200-thread pool is 100% blocked.

Set `management.server.port

8081`.

Configure `group.liveness.include

livenessState and group.readiness.include: readinessState,db`.

Step 3

Add startupProbe, readinessProbe, and livenessProbe in K8s spec.

Alternative 1

Use gRPC health probe or a dedicated lightweight Netty HTTP server for /livez.

Failure Mode 1

Exposing port 8081 on the public Ingress/Service instead of keeping it cluster-internal.

# application.yaml
management:
  server:
    port: 8081
  endpoint:
    health:
      probes:
        enabled: true
      group:
        liveness:
          include: livenessState
        readiness:
          include: readinessState,db
spring:
  datasource:
    hikari:
      connection-timeout: 1500
---
# Kubernetes Pod Spec
startupProbe:
  httpGet: { path: /actuator/health/liveness, port: 8081 }
  periodSeconds: 5
  failureThreshold: 18
readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 8081 }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2
  successThreshold: 2
livenessProbe:
  httpGet: { path: /actuator/health/liveness, port: 8081 }
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

Yes, for single-pod DB network.

Yes, for single-pod DB network splits, readiness should fail; however, if Postgres itself is down, all 24 pods failing readiness causes kube-proxy to drop all endpoints (503/connection refused) unless publishNotReadyAddresses or fast-fail 503s are preferred.

ChoiceWhat it buysWhat it costs
Trade-off 1Fast 503s keep pods in the Service pool so recovery is instantaneous when the DB lock clears.

Step 1

Distinguish local pod DB-pool exhaustion from global DB outage.

Step 2

Explain that returning HTTP 503 with Retry-After from warm pods is often cleaner than removing all 24 endpoints.

Alternative 1

Use Resilience4j circuit breaker to trip open after 50% DB failures.

Failure Mode 1

Removing all 24 endpoints causes kube-proxy to send TCP RST (connection refused) at the Ingress.

// Fail fast with 503 via Hikari timeout instead of blocking threads
@RestControllerAdvice
public class PoolOverloadHandler {
    @ExceptionHandler(SQLTransientConnectionException.class)
    public ResponseEntity<String> handlePoolExhausted() {
        return ResponseEntity.status(503).header("Retry-After", "2").body("DB pool saturated");
    }
}
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

Linux CFS CPU quota throttling.

Linux CFS CPU quota throttling (limits.cpu: 2) freezes the entire JVM process—including port 8081 threads—for up to 90+ ms per 100 ms period.

ChoiceWhat it buysWhat it costs
Trade-off 1Omitting `limits.cpu` eliminates CFS throttling pauses while `requests.cpu` still guarantees fair share under contention.

Step 1

Check container_cpu_cfs_throttled_periods_total.

Step 2

Remove limits.cpu (keep requests.cpu) or raise timeoutSeconds: 5 and limits.cpu headroom.

Alternative 1

Use timeoutSeconds: 5 + failureThreshold: 6 so transient 200 ms CFS pauses never accumulate 6 consecutive failures.

Failure Mode 1

Strict limits.cpu with multi-threaded JVM GC causes severe tail-latency and probe timeouts.

resources:
  requests:
    cpu: "2"
    memory: "4Gi"
  limits:
    memory: "4Gi"
    # Omit limits.cpu or set higher burst headroom to avoid CFS throttling
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Only unrecoverable internal failures.

Only unrecoverable internal failures: OutOfMemoryError (-XX:+ExitOnOutOfMemoryError exits directly without needing liveness), or an internal deadlock detected by ThreadMXBean.findDeadlockedThreads().

ChoiceWhat it buysWhat it costs
Trade-off 1`findDeadlockedThreads()` takes ~1-2 ms and inspects JVM monitor/ownable synchronizer cycles.

Use `-XX

+ExitOnOutOfMemoryError` for immediate OOM exit.

Step 2

Optionally check ThreadMXBean.findDeadlockedThreads() in a custom liveness indicator.

Alternative 1

Keep liveness strictly as a lightweight 200 OK ping and rely on -XX:+ExitOnOutOfMemoryError.

Failure Mode 1

Putting complex lock checks inside liveness without caching can itself add CPU overhead.

import java.lang.management.ManagementFactory;
import java.lang.management.ThreadMXBean;

public boolean isJvmDeadlocked() {
    ThreadMXBean bean = ManagementFactory.getThreadMXBean();
    long[] deadlocked = bean.findDeadlockedThreads();
    return deadlocked != null && deadlocked.length > 0;
}

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Separate /livez (dependency-free, dedicated lightweight thread/listener) from /readyz (dependency & pool capacity checks), add startupProbe, and size probe timeouts above GC/CFS pauses.
  • 02Trade-off tip: Fast 503s keep pods in the Service pool so recovery is instantaneous when the DB lock clears.
  • 03Watch out for: Strict limits.cpu with multi-threaded JVM GC causes severe tail-latency and probe timeouts.
Behavioral Progression · Designing Safe Liveness Readiness Probes

Designing Safe Liveness Readiness Probes — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Separate /livez (dependency-free, dedicated lightweight thread/listener) from /readyz (dependency & pool capacity checks), add startupProbe, and size probe timeouts above GC/CFS pauses.

Synthesis / One connected design

Putting Designing Safe Liveness & Readiness Probes Together

Separate `/livez` (dependency-free, dedicated lightweight thread/listener) from `/readyz` (dependency & pool capacity checks), add `startupProbe`, and size probe timeouts above GC/CFS pauses.

A candidate architecture, not a universal answer

Putting Designing Safe Liveness & Readiness Probes Together

Step 1
Step 2
Step 3
  1. Set management.server.port: 8081.
  2. Configure group.liveness.include: livenessState and group.readiness.include: readinessState,db.
  3. Add startupProbe, readinessProbe, and livenessProbe in K8s spec.
Linux CFS CPU quota throttling (`limits.cpu: 2`) freezes the entire JVM process—including port 8081 threads—for up to 90+ ms per 100 ms period.
Interview Readiness Check

Linux CFS CPU quota throttling (limits.cpu: 2) freezes the entire JVM process—including port 8081 threads—for up to 90+ ms per 100 ms period.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Designing Safe Liveness & Readiness Probes

Separate /livez (dependency-free, dedicated lightweight thread/listener) from /readyz (dependency & pool capacity checks), add startupProbe, and size probe timeouts above GC/CFS pauses.