During a mandatory Kubernetes 1.28 -> 1.29 EKS node-group security upgrade, the upgrade stalls for 6 hours and fails. Node drain logs show `Cannot evict pod as it would violate the pod's disruption budget` (HTTP 429 TooManyRequests from the Eviction API). Investigation reveals three broken PDBs across the cluster: (1) `redis-cache` has `replicas: 1` and `minAvailable: 1`, (2) `auth-service` has `replicas: 3`, `minAvailable: 3`, and (3) `billing-worker` has `minAvailable: 4` while its HPA scaled it down to 3 replicas during off-peak hours. Redesign the PDB standards, admission guardrails, and drain runbooks so cluster upgrades complete within SLA without risking availability.
From patterns to decisions.
During a mandatory Kubernetes 1.28 -> 1.29 EKS node-group security upgrade, the upgrade stalls for 6 hours and fails. Node drain logs show `Cannot evict pod as it would violate the pod's disruption budget` (HTTP 429 TooManyRequests from the Eviction API). Investigation reveals three broken PDBs across the cluster: (1) `redis-cache` has `replicas: 1` and `minAvailable: 1`, (2) `auth-service` has `replicas: 3`, `minAvailable: 3`, and (3) `billing-worker` has `minAvailable: 4` while its HPA scaled it down to 3 replicas during off-peak hours. Redesign the PDB standards, admission guardrails, and drain runbooks so cluster upgrades complete within SLA without risking availability.
Each PDB resulted in desiredHealthy.
Each PDB resulted in desiredHealthy >= currentHealthy, setting disruptionsAllowed=0 and causing the Kubernetes Eviction API to return HTTP 429 TooManyRequests.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | PDBs protect against concurrent multi-node drains but must always leave >=1 disruption allowed for single-node maintenance. |
Step 1
Run kubectl get pdb -A to inspect ALLOWED DISRUPTIONS column.
Step 2
Compare redis-cache (1-1=0), auth-service (3-3=0), and billing-worker (3-4=0).
Step 3
Identify that HPA scaling below a static minAvailable silently deadlocks node drains.
Alternative 1
Use Karpenter DoNotDisrupt annotations only for short-lived critical jobs.
Failure Mode 1
Assuming PDBs protect against OOMKills or EC2 hardware failures (they only govern voluntary Eviction API calls).
# Diagnostic command:
# kubectl get pdb -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,HEALTHY:.status.currentHealthy,ALLOWED:.status.disruptionsAllowed
Use maxUnavailable.
Use maxUnavailable: 1 (or 25%) with unhealthyPodEvictionPolicy: AlwaysAllow, and enforce a Kyverno ClusterPolicy rejecting maxUnavailable: 0 or minAvailable: 100%.
| Choice | What it buys | What it costs |
|---|---|---|
| maxUnavailable | 1 guarantees safe node-by-node upgrades across any HPA replica count >= 2. |
Step 1
Update auth-service and billing-worker PDBs to maxUnavailable: 1.
Add unhealthyPodEvictionPolicy
AlwaysAllow (K8s 1.27+).
Step 3
Deploy Kyverno ClusterPolicy in Enforce mode.
Alternative 1
Use OPA Gatekeeper ConstraintTemplate for the same admission check.
Using maxUnavailable
25% on a 2-replica service rounds down to 0 in older K8s versions unless min 1 is used.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: billing-worker-pdb
spec:
maxUnavailable: 1
unhealthyPodEvictionPolicy: AlwaysAllow
selector:
matchLabels:
app: billing-worker
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: block-deadlocking-pdb
spec:
validationFailureAction: Enforce
rules:
- name: disallow-zero-disruptions
match:
any:
- resources:
kinds: ["PodDisruptionBudget"]
validate:
message: "PDB must not set maxUnavailable=0 or minAvailable=100%."
deny:
conditions:
any:
- key: "{{ request.object.spec.maxUnavailable || '' }}"
operator: Equals
value: 0
- key: "{{ request.object.spec.minAvailable || '' }}"
operator: Equals
value: "100%"
Implementation & Guardrails (3).
Use maxUnavailable for stateless/HPA workloads so scaling down never deadlocks drains; use minAvailable only for fixed-size quorum sets (e.g., minAvailable: 2 on 3-node etcd), and omit PDBs on singletons.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Omitting PDB on a singleton allows brief 10-30s restart during upgrades instead of blocking the entire node upgrade for 6 hours. |
Use maxUnavailable
1 (or 25%) for all Deployments with HPA.
Use minAvailable
2 for 3-node quorum StatefulSets.
Scale singletons to replicas
2 or remove their PDB.
Alternative 1
Use Blue/Green node-group replacement for singletons during scheduled maintenance windows.
Failure Mode 1
Combining minAvailable and maxUnavailable in one PDB is rejected by the Kubernetes API.
# 3-node quorum StatefulSet (e.g., Zookeeper/etcd): allows 1 node down at a time
spec:
minAvailable: 2
selector:
matchLabels:
app: zookeeper
Implementation & Guardrails (4).
By default (IfHealthyBudget), a CrashLooping pod reduces currentHealthy below desiredHealthy, causing K8s to block eviction of even the broken pod itself. AlwaysAllow lets broken pods be evicted.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Unblocks node upgrades immediately without weakening availability guarantees for healthy pods. |
Step 1
Verify cluster version >= 1.27.
Set unhealthyPodEvictionPolicy
AlwaysAllow on PDBs.
Step 3
Confirm drain evicts unhealthy pods without violating healthy pod budgets.
Alternative 1
Manually delete the CrashLoopBackOff pod before draining the node.
Failure Mode 1
Pre-1.27 clusters reject the unhealthyPodEvictionPolicy field.
spec:
maxUnavailable: 1
unhealthyPodEvictionPolicy: AlwaysAllow
Implementation & Guardrails (5).
Alert on kube_poddisruptionbudget_status_pod_disruptions_allowed == 0 sustained for >15 minutes via kube-state-metrics.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | 15-minute window filters out brief 30s rolling-update transitions while catching chronic misconfigurations. |
Step 1
Ensure kube-state-metrics scrapes policy/v1 PodDisruptionBudgets.
Step 2
Create alert rule for disruptions_allowed == 0 for 15m.
Step 3
Include namespace and poddisruptionbudget labels in runbook link.
Alternative 1
Run a pre-upgrade CI/script check listing all PDBs with disruptionsAllowed == 0.
Alerting with for
0m causes noisy false positives during normal pod restarts.
- alert: KubernetesPdbZeroDisruptionsAllowed
expr: kube_poddisruptionbudget_status_pod_disruptions_allowed == 0
for: 15m
labels:
severity: warning
annotations:
summary: "PDB {{ $labels.namespace }}/{{ $labels.poddisruptionbudget }} blocks node drains"
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Never set minAvailable equal to replicas on single- or fixed-replica workloads; use maxUnavailable: 1 (or >=25%) with replicas >= 2, and monitor kube_poddisruptionbudget_status_pod_disruptions_allowed == 0.
- 02Trade-off tip: Omitting PDB on a singleton allows brief 10-30s restart during upgrades instead of blocking the entire node upgrade for 6 hours.
- 03Watch out for: Pre-1.27 clusters reject the unhealthyPodEvictionPolicy field.
Kubernetes Pdb Safe Node Drains — Normal → Load / Fault → Recovery
Never set minAvailable equal to replicas on single- or fixed-replica workloads; use maxUnavailable: 1 (or >=25%) with replicas >= 2, and monitor kubepoddisruptionbudgetstatuspoddisruptionsallowed == 0.
Putting Kubernetes Pod Disruption Budgets & Safe Node Drains Together
Never set minAvailable equal to replicas on single- or fixed-replica workloads; use maxUnavailable: 1 (or >=25%) with replicas >= 2, and monitor kube_poddisruptionbudget_status_pod_disruptions_allowed == 0.
Putting Kubernetes Pod Disruption Budgets & Safe Node Drains Together
- Update auth-service and billing-worker PDBs to maxUnavailable: 1.
- Add unhealthyPodEvictionPolicy: AlwaysAllow (K8s 1.27+).
- Deploy Kyverno ClusterPolicy in Enforce mode.
Interview Readiness Check
By default (IfHealthyBudget), a CrashLooping pod reduces currentHealthy below desiredHealthy, causing K8s to block eviction of even the broken pod itself. AlwaysAllow lets broken pods be evicted.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Kubernetes Pod Disruption Budgets & Safe Node Drains
Never set minAvailable equal to replicas on single- or fixed-replica workloads; use maxUnavailable: 1 (or >=25%) with replicas >= 2, and monitor kube_poddisruptionbudget_status_pod_disruptions_allowed == 0.
