A latency-sensitive Java/Netty payment gateway running in Kubernetes (`requests.cpu: '500m'`, `limits.cpu: '2'`) on 64-vCPU nodes has a median latency of 4ms, but p99 latency spikes to 95-120ms every few seconds. Node CPU utilization is only 35%, yet `container_cpu_cfs_throttled_periods_total` is 45% and `nr_throttled` climbs rapidly. The JVM sees all 64 host cores via `/proc/cpuinfo` (or defaults) and spawns 64 Netty/ForkJoin/G1GC threads. Diagnose the Linux CFS bandwidth mechanism causing the 95ms freezes and fix the Kubernetes and runtime configuration.
From patterns to decisions.
A latency-sensitive Java/Netty payment gateway running in Kubernetes (`requests.cpu: '500m'`, `limits.cpu: '2'`) on 64-vCPU nodes has a median latency of 4ms, but p99 latency spikes to 95-120ms every few seconds. Node CPU utilization is only 35%, yet `container_cpu_cfs_throttled_periods_total` is 45% and `nr_throttled` climbs rapidly. The JVM sees all 64 host cores via `/proc/cpuinfo` (or defaults) and spawns 64 Netty/ForkJoin/G1GC threads. Diagnose the Linux CFS bandwidth mechanism causing the 95ms freezes and fix the Kubernetes and runtime configuration.
Linux CFS gives a 2.
Linux CFS gives a 2 CPU pod 200ms of CPU time per 100ms wall-clock period. 16 parallel threads consume 200ms in 12.5ms, freezing the entire pod for the remaining 87.5ms.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | 5-minute average CPU hides 100ms-scale CFS throttling completely. |
Step 1
Compute cpu.cfs_quota_us (200,000) / cpu.cfs_period_us (100,000).
Step 2
Divide 200ms by 16 active threads to show 12.5ms execution + 87.5ms freeze.
Alternative 1
Inspect /sys/fs/cgroup/cpu.stat (nr_throttled, throttled_usec).
Failure Mode 1
GC pauses and Netty event-loop stalls cascade into upstream timeouts.
# Anti-pattern: bursty multi-threaded JVM with low requests and hard 2-core limit
resources:
requests:
cpu: "500m"
limits:
cpu: "2" # 200ms quota per 100ms period -> severe throttling!
Right-size requests.
Right-size requests.cpu: 2, remove limits.cpu (or use Guaranteed static pinning), and pass -XX:ActiveProcessorCount=2 to the JVM.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Zero CFS throttling; pod is guaranteed 2 cores via `cpu.weight` under contention and can burst when host is idle. |
Set `requests.cpu
"2" and remove limits.cpu` for zero CFS throttling.
Add `JAVA_TOOL_OPTIONS` with `-XX
ActiveProcessorCount=2` and bounded GC threads.
Alternative 1
Set limits.cpu: "2" + Guaranteed QoS on a node pool with --cpu-manager-policy=static.
Leaving `requests.cpu
500m without limits.cpu` starves the pod during node contention.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-gateway
spec:
template:
spec:
containers:
- name: gateway
image: payment-gateway:2.4.1
env:
- name: JAVA_TOOL_OPTIONS
value: "-XX:ActiveProcessorCount=2 -XX:ParallelGCThreads=2 -XX:ConcGCThreads=1"
resources:
requests:
cpu: "2"
memory: "4Gi"
limits:
memory: "4Gi"
# cpu limit intentionally omitted to prevent CFS throttling freezes
Use either (1) Guaranteed QoS.
Use either (1) Guaranteed QoS with integer CPUs + kubelet --cpu-manager-policy=static (bypasses CFS quota via cpuset), or (2) CustomCPUCFSQuotaPeriod reducing cpuCFSQuotaPeriod from 100ms to 5ms.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | `static` CPU Manager requires integer CPU requests (`2`, not `1500m`) and `Guaranteed` QoS (`requests == limits` for both CPU and memory). |
Configure `cpuManagerPolicy
static` in KubeletConfiguration for exclusive core pinning.
Alternatively set `cpuCFSQuotaPeriod
5ms` so any freeze is <= 3.75ms.
Alternative 1
Keep 100ms period and drop limits.cpu for Burstable pods.
Failure Mode 1
Fractional CPU (1500m) silently falls back to the shared CFS pool even when cpuManagerPolicy: static is enabled.
# KubeletConfiguration for strict limits without 90ms freezes
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuManagerPolicy: static
cpuCFSQuota: true
cpuCFSQuotaPeriod: 5ms
In Go, use uber-go/automaxprocs (reads.
In Go, use uber-go/automaxprocs (reads cgroup CPU quota and sets GOMAXPROCS=2 instead of 64). In Python/Gunicorn, size --workers and OMP_NUM_THREADS to the CPU request.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Dynamic cgroup inspection prevents worker over-allocation when pod CPU requests are resized. |
Step 1
Avoid reading os.cpu_count() (returns 64 host cores) when sizing Gunicorn/Uvicorn/ThreadPoolExecutor.
Step 2
Read /sys/fs/cgroup/cpu.max in Python to cap worker/thread pools dynamically.
Alternative 1
Pass WEB_CONCURRENCY=2 and OMP_NUM_THREADS=2 explicitly via Kubernetes Downward API.
Failure Mode 1
os.cpu_count() in Python ignores cgroup v2 cpu.max and returns 64.
import math
from pathlib import Path
def cgroup_cpu_limit(default: int = 2) -> int:
cpu_max = Path("/sys/fs/cgroup/cpu.max")
if cpu_max.exists():
quota, period = cpu_max.read_text().strip().split()
if quota != "max":
return max(1, math.ceil(int(quota) / int(period)))
return default
Alert on the ratio of.
Alert on the ratio of throttled periods (container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total > 0.05) and total throttled time (container_cpu_cfs_throttled_seconds_total).
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Catches sub-second throttling freezes immediately regardless of low 5-minute average CPU. |
Step 1
Monitor both throttled_periods ratio (>5%) and throttled_seconds rate.
Step 2
Correlate spikes directly against service http_request_duration_seconds p99.
Alternative 1
Read /sys/fs/cgroup/cpu.stat throttled_usec directly via OpenTelemetry hostmetrics.
Failure Mode 1
Division by zero on idle containers with container_cpu_cfs_periods_total == 0 unless filtered.
# PrometheusRule for CFS CPU throttling
- alert: HighContainerCPUThrottling
expr: |
(
rate(container_cpu_cfs_throttled_periods_total{container!=""}[2m])
/
rate(container_cpu_cfs_periods_total{container!=""}[2m])
) > 0.05
for: 5m
labels:
severity: warning
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Align runtime thread pools (
ActiveProcessorCount/GOMAXPROCS) to the CPU quota, useGuaranteedQoS with integer CPUs (statickubelet CPU Manager) or removelimits.cpuon latency-critical pods, and shortencfs_period_usif limits are mandatory. - 02Trade-off tip:
staticCPU Manager requires integer CPU requests (2, not1500m) andGuaranteedQoS (requests == limitsfor both CPU and memory). - 03Watch out for:
os.cpu_count()in Python ignores cgroup v2cpu.maxand returns 64.
Fixing Kubernetes Cfs Cpu Throttling — Normal → Load / Fault → Recovery
Align runtime thread pools (ActiveProcessorCount / GOMAXPROCS) to the CPU quota, use Guaranteed QoS with integer CPUs (static kubelet CPU Manager) or remove limits.cpu on latency-critical pods, and shorten cfsperiodus if limits are mandatory.
Putting Fixing Kubernetes CFS CPU Throttling Freezes Together
Align runtime thread pools (`ActiveProcessorCount` / `GOMAXPROCS`) to the CPU quota, use `Guaranteed` QoS with integer CPUs (`static` kubelet CPU Manager) or remove `limits.cpu` on latency-critical pods, and shorten `cfs_period_us` if limits are mandatory.
Putting Fixing Kubernetes CFS CPU Throttling Freezes Together
- Set
requests.cpu: "2"and removelimits.cpufor zero CFS throttling. - Add
JAVA_TOOL_OPTIONSwith-XX:ActiveProcessorCount=2and bounded GC threads.
Interview Readiness Check
In Go, use uber-go/automaxprocs (reads cgroup CPU quota and sets GOMAXPROCS=2 instead of 64). In Python/Gunicorn, size --workers and OMP_NUM_THREADS to the CPU request.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Fixing Kubernetes CFS CPU Throttling Freezes
Align runtime thread pools (ActiveProcessorCount / GOMAXPROCS) to the CPU quota, use Guaranteed QoS with integer CPUs (static kubelet CPU Manager) or remove limits.cpu on latency-critical pods, and shorten cfs_period_us if limits are mandatory.
