Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Kubernetes Pod Disruption Budgets
& Safe Node Drains.

During a mandatory Kubernetes 1.28 -> 1.29 EKS node-group security upgrade, the upgrade stalls for 6 hours and fails. Node drain logs show Cannot evict pod as it would violate the pod's disruption budget (HTTP 429 TooManyRequests from the Eviction API). Investigation reveals three broken PDBs across the cluster: (1) redis-cache has replicas: 1 and minAvailable: 1, (2) auth-service has replicas: 3, minAvailable: 3, and (3) billing-worker has minAvailable: 4 while its HPA scaled it down to 3 replicas during off-peak hours. Redesign the PDB standards, admission guardrails, and drain runbooks so cluster upgrades complete within SLA without risking availability.
Start the journey5 stages
Loading diagram…
Kubernetes Pod Disruption Budgets & Safe Node Drains architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

During a mandatory Kubernetes 1.28 -> 1.29 EKS node-group security upgrade, the upgrade stalls for 6 hours and fails. Node drain logs show `Cannot evict pod as it would violate the pod's disruption budget` (HTTP 429 TooManyRequests from the Eviction API). Investigation reveals three broken PDBs across the cluster: (1) `redis-cache` has `replicas: 1` and `minAvailable: 1`, (2) `auth-service` has `replicas: 3`, `minAvailable: 3`, and (3) `billing-worker` has `minAvailable: 4` while its HPA scaled it down to 3 replicas during off-peak hours. Redesign the PDB standards, admission guardrails, and drain runbooks so cluster upgrades complete within SLA without risking availability.

From patterns to decisions.

During a mandatory Kubernetes 1.28 -> 1.29 EKS node-group security upgrade, the upgrade stalls for 6 hours and fails. Node drain logs show `Cannot evict pod as it would violate the pod's disruption budget` (HTTP 429 TooManyRequests from the Eviction API). Investigation reveals three broken PDBs across the cluster: (1) `redis-cache` has `replicas: 1` and `minAvailable: 1`, (2) `auth-service` has `replicas: 3`, `minAvailable: 3`, and (3) `billing-worker` has `minAvailable: 4` while its HPA scaled it down to 3 replicas during off-peak hours. Redesign the PDB standards, admission guardrails, and drain runbooks so cluster upgrades complete within SLA without risking availability.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

Each PDB resulted in desiredHealthy.

Each PDB resulted in desiredHealthy >= currentHealthy, setting disruptionsAllowed=0 and causing the Kubernetes Eviction API to return HTTP 429 TooManyRequests.

ChoiceWhat it buysWhat it costs
Trade-off 1PDBs protect against concurrent multi-node drains but must always leave >=1 disruption allowed for single-node maintenance.

Step 1

Run kubectl get pdb -A to inspect ALLOWED DISRUPTIONS column.

Step 2

Compare redis-cache (1-1=0), auth-service (3-3=0), and billing-worker (3-4=0).

Step 3

Identify that HPA scaling below a static minAvailable silently deadlocks node drains.

Alternative 1

Use Karpenter DoNotDisrupt annotations only for short-lived critical jobs.

Failure Mode 1

Assuming PDBs protect against OOMKills or EC2 hardware failures (they only govern voluntary Eviction API calls).

# Diagnostic command:
# kubectl get pdb -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,HEALTHY:.status.currentHealthy,ALLOWED:.status.disruptionsAllowed
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Use maxUnavailable.

Use maxUnavailable: 1 (or 25%) with unhealthyPodEvictionPolicy: AlwaysAllow, and enforce a Kyverno ClusterPolicy rejecting maxUnavailable: 0 or minAvailable: 100%.

ChoiceWhat it buysWhat it costs
maxUnavailable1 guarantees safe node-by-node upgrades across any HPA replica count >= 2.

Step 1

Update auth-service and billing-worker PDBs to maxUnavailable: 1.

Add unhealthyPodEvictionPolicy

AlwaysAllow (K8s 1.27+).

Step 3

Deploy Kyverno ClusterPolicy in Enforce mode.

Alternative 1

Use OPA Gatekeeper ConstraintTemplate for the same admission check.

Using maxUnavailable

25% on a 2-replica service rounds down to 0 in older K8s versions unless min 1 is used.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: billing-worker-pdb
spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app: billing-worker
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-deadlocking-pdb
spec:
  validationFailureAction: Enforce
  rules:
    - name: disallow-zero-disruptions
      match:
        any:
          - resources:
              kinds: ["PodDisruptionBudget"]
      validate:
        message: "PDB must not set maxUnavailable=0 or minAvailable=100%."
        deny:
          conditions:
            any:
              - key: "{{ request.object.spec.maxUnavailable || '' }}"
                operator: Equals
                value: 0
              - key: "{{ request.object.spec.minAvailable || '' }}"
                operator: Equals
                value: "100%"
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (3).

Use maxUnavailable for stateless/HPA workloads so scaling down never deadlocks drains; use minAvailable only for fixed-size quorum sets (e.g., minAvailable: 2 on 3-node etcd), and omit PDBs on singletons.

ChoiceWhat it buysWhat it costs
Trade-off 1Omitting PDB on a singleton allows brief 10-30s restart during upgrades instead of blocking the entire node upgrade for 6 hours.

Use maxUnavailable

1 (or 25%) for all Deployments with HPA.

Use minAvailable

2 for 3-node quorum StatefulSets.

Scale singletons to replicas

2 or remove their PDB.

Alternative 1

Use Blue/Green node-group replacement for singletons during scheduled maintenance windows.

Failure Mode 1

Combining minAvailable and maxUnavailable in one PDB is rejected by the Kubernetes API.

# 3-node quorum StatefulSet (e.g., Zookeeper/etcd): allows 1 node down at a time
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: zookeeper
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (4).

By default (IfHealthyBudget), a CrashLooping pod reduces currentHealthy below desiredHealthy, causing K8s to block eviction of even the broken pod itself. AlwaysAllow lets broken pods be evicted.

ChoiceWhat it buysWhat it costs
Trade-off 1Unblocks node upgrades immediately without weakening availability guarantees for healthy pods.

Step 1

Verify cluster version >= 1.27.

Set unhealthyPodEvictionPolicy

AlwaysAllow on PDBs.

Step 3

Confirm drain evicts unhealthy pods without violating healthy pod budgets.

Alternative 1

Manually delete the CrashLoopBackOff pod before draining the node.

Failure Mode 1

Pre-1.27 clusters reject the unhealthyPodEvictionPolicy field.

spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Implementation & Guardrails (5).

Alert on kube_poddisruptionbudget_status_pod_disruptions_allowed == 0 sustained for >15 minutes via kube-state-metrics.

ChoiceWhat it buysWhat it costs
Trade-off 115-minute window filters out brief 30s rolling-update transitions while catching chronic misconfigurations.

Step 1

Ensure kube-state-metrics scrapes policy/v1 PodDisruptionBudgets.

Step 2

Create alert rule for disruptions_allowed == 0 for 15m.

Step 3

Include namespace and poddisruptionbudget labels in runbook link.

Alternative 1

Run a pre-upgrade CI/script check listing all PDBs with disruptionsAllowed == 0.

Alerting with for

0m causes noisy false positives during normal pod restarts.

- alert: KubernetesPdbZeroDisruptionsAllowed
  expr: kube_poddisruptionbudget_status_pod_disruptions_allowed == 0
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "PDB {{ $labels.namespace }}/{{ $labels.poddisruptionbudget }} blocks node drains"

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Never set minAvailable equal to replicas on single- or fixed-replica workloads; use maxUnavailable: 1 (or >=25%) with replicas >= 2, and monitor kube_poddisruptionbudget_status_pod_disruptions_allowed == 0.
  • 02Trade-off tip: Omitting PDB on a singleton allows brief 10-30s restart during upgrades instead of blocking the entire node upgrade for 6 hours.
  • 03Watch out for: Pre-1.27 clusters reject the unhealthyPodEvictionPolicy field.
Behavioral Progression · Kubernetes Pdb Safe Node Drains

Kubernetes Pdb Safe Node Drains — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Never set minAvailable equal to replicas on single- or fixed-replica workloads; use maxUnavailable: 1 (or >=25%) with replicas >= 2, and monitor kubepoddisruptionbudgetstatuspoddisruptionsallowed == 0.

Synthesis / One connected design

Putting Kubernetes Pod Disruption Budgets & Safe Node Drains Together

Never set minAvailable equal to replicas on single- or fixed-replica workloads; use maxUnavailable: 1 (or >=25%) with replicas >= 2, and monitor kube_poddisruptionbudget_status_pod_disruptions_allowed == 0.

A candidate architecture, not a universal answer

Putting Kubernetes Pod Disruption Budgets & Safe Node Drains Together

Step 1
Step 2
Step 3
  1. Update auth-service and billing-worker PDBs to maxUnavailable: 1.
  2. Add unhealthyPodEvictionPolicy: AlwaysAllow (K8s 1.27+).
  3. Deploy Kyverno ClusterPolicy in Enforce mode.
By default (IfHealthyBudget), a CrashLooping pod reduces currentHealthy below desiredHealthy, causing K8s to block eviction of even the broken pod itself. AlwaysAllow lets broken pods be evicted.
Interview Readiness Check

By default (IfHealthyBudget), a CrashLooping pod reduces currentHealthy below desiredHealthy, causing K8s to block eviction of even the broken pod itself. AlwaysAllow lets broken pods be evicted.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Kubernetes Pod Disruption Budgets & Safe Node Drains

Never set minAvailable equal to replicas on single- or fixed-replica workloads; use maxUnavailable: 1 (or >=25%) with replicas >= 2, and monitor kube_poddisruptionbudget_status_pod_disruptions_allowed == 0.