Your microservices cluster (120 nodes, `40,000 req/s` total outbound HTTP/gRPC calls to external APIs like `api.stripe.com`, `s3.us-east-1.amazonaws.com`, and internal services) experiences intermittent `5-second` DNS timeouts (`i/o timeout` / `UnknownHostException`). Inspecting CoreDNS (6 replicas) shows `280,000 QPS` of `NXDOMAIN` responses for `api.stripe.com.prod.svc.cluster.local`, `api.stripe.com.svc.cluster.local`, and `api.stripe.com.cluster.local`, saturating CoreDNS CPU and `nf_conntrack` UDP tables.
TL;DR30-second simplified summary▾
Your microservices cluster (120 nodes, 40,000 req/s total outbound HTTP/gRPC calls to external APIs like api.stripe.com, s3.us-east-1.amazonaws.com, and internal services) experiences intermittent 5-second DNS timeouts (i/o timeout / UnknownHostException). Inspecting CoreDNS (6 replicas) shows 280,000 QPS of NXDOMAIN responses for api.stripe.com.prod.svc.cluster.local, api.stripe.com.svc.cluster.local, and api.stripe.com.cluster.local, saturating CoreDNS CPU and nf_conntrack UDP tables.
Deploy NodeLocal DNSCache (169.254.20.10 with TCP upstream), lower pod ndots to 2 (or use trailing dots api.stripe.com.), and tune CoreDNS cache + autopath.
From patterns to decisions.
Your microservices cluster (120 nodes, `40,000 req/s` total outbound HTTP/gRPC calls to external APIs like `api.stripe.com`, `s3.us-east-1.amazonaws.com`, and internal services) experiences intermittent `5-second` DNS timeouts (`i/o timeout` / `UnknownHostException`). Inspecting CoreDNS (6 replicas) shows `280,000 QPS` of `NXDOMAIN` responses for `api.stripe.com.prod.svc.cluster.local`, `api.stripe.com.svc.cluster.local`, and `api.stripe.com.cluster.local`, saturating CoreDNS CPU and `nf_conntrack` UDP tables.
Implementation & Guardrails (1).
api.stripe.com has 2 dots (< 5), so glibc appends all 4 search suffixes first, sending 8 A/AAAA NXDOMAIN queries before the real one. Parallel A+AAAA UDP packets race in nf_conntrack, causing 5s retries.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Trailing-dot FQDNs (`api.stripe.com.`) bypass search paths with zero cluster config, but break TLS SNI/Host headers in some HTTP clients if not stripped. |
Step 1
Inspect /etc/resolv.conf inside a running pod.
Step 2
Enable CoreDNS log briefly or inspect coredns_dns_responses_total{rcode='NXDOMAIN'}.
Step 3
Check conntrack -S for insert_failed / UDP drops.
Alternative 1
Use coredns autopath plugin for pods without NodeLocal DNSCache.
Failure Mode 1
Using api.stripe.com. in HTTP Host headers can cause 400 Bad Request on strict CDNs.
def count_dns_queries(hostname: str, ndots: int, search_domains: int, dual_stack: bool = True) -> int:
dots = hostname.rstrip(".").count(".")
per_try = 2 if dual_stack else 1 # A + AAAA
if hostname.endswith(".") or dots >= ndots:
return per_try
return (search_domains + 1) * per_try
Mutate application pods to ndots.
Mutate application pods to ndots:2 + single-request-reopen via Kyverno, and run NodeLocal DNSCache with force_tcp upstream to CoreDNS.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Mutating `dnsConfig` applies cleanly to all new pods without requiring code changes across 50+ repos. |
Step 1
Deploy node-local-dns DaemonSet on 169.254.20.10 and point kubelet --cluster-dns to it.
Apply Kyverno mutation policy setting `ndots
2`.
Step 3
Verify NXDOMAIN rate drops >80% in Prometheus.
Alternative 1
Set dnsConfig directly in shared Helm chart templates.
Setting `ndots
1 instead of 2 breaks service.namespace` (1-dot) cross-namespace lookups.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: inject-ndots-2
spec:
rules:
- name: set-ndots-2
match:
any:
- resources:
kinds: [Pod]
namespaces: ["!kube-system"]
mutate:
patchStrategicMerge:
spec:
dnsConfig:
options:
- name: ndots
value: "2"
- name: single-request-reopen
Implementation & Guardrails (3).
ndots:2 still expands 0-dot (redis) and 1-dot (auth.payments) internal names via .svc.cluster.local, whereas ndots:1 sends auth.payments to root DNS first. force_tcp avoids UDP conntrack drops between nodes and CoreDNS.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | TCP upstream adds a tiny connection-setup cost amortized over persistent connections, while eliminating UDP packet loss. |
Test `dots < ndots`
for ndots=2, 0 < 2 and 1 < 2 (search path used); 2 >= 2 (external root first).
Step 2
Use NOTRACK on 169.254.20.10 for pod-to-NodeLocal UDP.
Step 3
Use force_tcp from NodeLocal to CoreDNS so kube-proxy NAT never drops UDP.
Alternative 1
Use Cilium eBPF socket-level load balancing to remove iptables UDP conntrack races.
Using `ndots
1 causes auth.payments to leak internal namespace names to external DNS servers and return NXDOMAIN`.
def resolves_internally_first(name: str, ndots: int = 2) -> bool:
# Names with fewer than ndots try .svc.cluster.local suffixes first
return name.count(".") < ndots
Implementation & Guardrails (4).
Enable serve_stale in CoreDNS/NodeLocal (cache 30 { serve_stale 5m immediate }) so transient upstream VPC DNS throttling serves slightly stale records instead of SERVFAIL.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | `serve_stale` prioritizes availability over strict TTL freshness during an upstream DNS outage. |
Step 1
Add serve_stale 5m immediate to CoreDNS cache plugin.
Step 2
Configure max_concurrent and health / ready timeouts.
Step 3
Increase VPC resolver / Route 53 Resolver inbound endpoint capacity if throttled.
Alternative 1
Run redundant upstream resolvers (e.g., VPC DNS + Unbound fallback).
Failure Mode 1
Setting serve_stale for too long (e.g., 24h) can pin traffic to a decommissioned IP after an outage.
# Corefile cache block with serve_stale
.:53 {
cache 30 {
success 9984 30
denial 9984 15
serve_stale 5m immediate
}
forward . /etc/resolv.conf {
max_concurrent 1000
}
}
Implementation & Guardrails (5).
Track coredns_dns_responses_total{rcode='NXDOMAIN'} / total and node_nf_conntrack_stat_drop, alerting if NXDOMAIN ratio exceeds 40% or p99 DNS latency exceeds 20 ms.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Monitoring `NXDOMAIN` ratio catches misconfigured `ndots:5` workloads before they saturate CoreDNS. |
Scrape CoreDNS `
9153 and NodeLocal :9253` metrics in Prometheus.
Step 2
Alert on NXDOMAIN ratio > 40% (indicates ndots:5 regression).
Step 3
Alert on coredns_dns_request_duration_seconds p99 > 20 ms.
Alternative 1
Use Hubble / eBPF DNS observability to attribute top NXDOMAIN callers by pod label.
Failure Mode 1
Averaging CoreDNS latency across all nodes hides single-node conntrack drops.
def check_dns_health(nxdomain_qps: float, total_qps: float, p99_ms: float) -> str:
nx_ratio = nxdomain_qps / max(1.0, total_qps)
if p99_ms > 50.0:
return "CRITICAL_DNS_LATENCY"
if nx_ratio > 0.40:
return "WARN_NDOTS_AMPLIFICATION"
return "HEALTHY"
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Deploy
NodeLocal DNSCache(169.254.20.10with TCP upstream), lower podndotsto2(or use trailing dotsapi.stripe.com.), and tune CoreDNScache+autopath. - 02Trade-off tip: TCP upstream adds a tiny connection-setup cost amortized over persistent connections, while eliminating UDP packet loss.
- 03Watch out for: Setting
serve_stalefor too long (e.g., 24h) can pin traffic to a decommissioned IP after an outage.
Putting Fixing CoreDNS Query Flooding from ndots:5 Together
Deploy `NodeLocal DNSCache` (`169.254.20.10` with TCP upstream), lower pod `ndots` to `2` (or use trailing dots `api.stripe.com.`), and tune CoreDNS `cache` + `autopath`.
Putting Fixing CoreDNS Query Flooding from ndots:5 Together
- Deploy
node-local-dnsDaemonSet on169.254.20.10and pointkubelet--cluster-dnsto it. - Apply Kyverno mutation policy setting
ndots: 2. - Verify
NXDOMAINrate drops >80% in Prometheus.
Interview Readiness Check
Enable serve_stale in CoreDNS/NodeLocal (cache 30 { serve_stale 5m immediate }) so transient upstream VPC DNS throttling serves slightly stale records instead of SERVFAIL.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Practice: Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Fixing CoreDNS Query Flooding from ndots:5
Deploy NodeLocal DNSCache (169.254.20.10 with TCP upstream), lower pod ndots to 2 (or use trailing dots api.stripe.com.), and tune CoreDNS cache + autopath.
