Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Fixing CoreDNS Query
Flooding from ndots:5.

Your microservices cluster (120 nodes, 40,000 req/s total outbound HTTP/gRPC calls to external APIs like api.stripe.com, s3.us-east-1.amazonaws.com, and internal services) experiences intermittent 5-second DNS timeouts (i/o timeout / UnknownHostException). Inspecting CoreDNS (6 replicas) shows 280,000 QPS of NXDOMAIN responses for api.stripe.com.prod.svc.cluster.local, api.stripe.com.svc.cluster.local, and api.stripe.com.cluster.local, saturating CoreDNS CPU and nf_conntrack UDP tables.
Start the journey5 stages
Loading diagram…
Fixing CoreDNS Query Flooding from ndots:5 architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your microservices cluster (120 nodes, `40,000 req/s` total outbound HTTP/gRPC calls to external APIs like `api.stripe.com`, `s3.us-east-1.amazonaws.com`, and internal services) experiences intermittent `5-second` DNS timeouts (`i/o timeout` / `UnknownHostException`). Inspecting CoreDNS (6 replicas) shows `280,000 QPS` of `NXDOMAIN` responses for `api.stripe.com.prod.svc.cluster.local`, `api.stripe.com.svc.cluster.local`, and `api.stripe.com.cluster.local`, saturating CoreDNS CPU and `nf_conntrack` UDP tables.

TL;DR30-second simplified summary
▾

Your microservices cluster (120 nodes, 40,000 req/s total outbound HTTP/gRPC calls to external APIs like api.stripe.com, s3.us-east-1.amazonaws.com, and internal services) experiences intermittent 5-second DNS timeouts (i/o timeout / UnknownHostException). Inspecting CoreDNS (6 replicas) shows 280,000 QPS of NXDOMAIN responses for api.stripe.com.prod.svc.cluster.local, api.stripe.com.svc.cluster.local, and api.stripe.com.cluster.local, saturating CoreDNS CPU and nf_conntrack UDP tables.

Deploy NodeLocal DNSCache (169.254.20.10 with TCP upstream), lower pod ndots to 2 (or use trailing dots api.stripe.com.), and tune CoreDNS cache + autopath.

From patterns to decisions.

Your microservices cluster (120 nodes, `40,000 req/s` total outbound HTTP/gRPC calls to external APIs like `api.stripe.com`, `s3.us-east-1.amazonaws.com`, and internal services) experiences intermittent `5-second` DNS timeouts (`i/o timeout` / `UnknownHostException`). Inspecting CoreDNS (6 replicas) shows `280,000 QPS` of `NXDOMAIN` responses for `api.stripe.com.prod.svc.cluster.local`, `api.stripe.com.svc.cluster.local`, and `api.stripe.com.cluster.local`, saturating CoreDNS CPU and `nf_conntrack` UDP tables.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

Implementation & Guardrails (1).

api.stripe.com has 2 dots (< 5), so glibc appends all 4 search suffixes first, sending 8 A/AAAA NXDOMAIN queries before the real one. Parallel A+AAAA UDP packets race in nf_conntrack, causing 5s retries.

ChoiceWhat it buysWhat it costs
Trade-off 1Trailing-dot FQDNs (`api.stripe.com.`) bypass search paths with zero cluster config, but break TLS SNI/Host headers in some HTTP clients if not stripped.

Step 1

Inspect /etc/resolv.conf inside a running pod.

Step 2

Enable CoreDNS log briefly or inspect coredns_dns_responses_total{rcode='NXDOMAIN'}.

Step 3

Check conntrack -S for insert_failed / UDP drops.

Alternative 1

Use coredns autopath plugin for pods without NodeLocal DNSCache.

Failure Mode 1

Using api.stripe.com. in HTTP Host headers can cause 400 Bad Request on strict CDNs.

def count_dns_queries(hostname: str, ndots: int, search_domains: int, dual_stack: bool = True) -> int:
    dots = hostname.rstrip(".").count(".")
    per_try = 2 if dual_stack else 1  # A + AAAA
    if hostname.endswith(".") or dots >= ndots:
        return per_try
    return (search_domains + 1) * per_try
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Mutate application pods to ndots.

Mutate application pods to ndots:2 + single-request-reopen via Kyverno, and run NodeLocal DNSCache with force_tcp upstream to CoreDNS.

ChoiceWhat it buysWhat it costs
Trade-off 1Mutating `dnsConfig` applies cleanly to all new pods without requiring code changes across 50+ repos.

Step 1

Deploy node-local-dns DaemonSet on 169.254.20.10 and point kubelet --cluster-dns to it.

Apply Kyverno mutation policy setting `ndots

2`.

Step 3

Verify NXDOMAIN rate drops >80% in Prometheus.

Alternative 1

Set dnsConfig directly in shared Helm chart templates.

Setting `ndots

1 instead of 2 breaks service.namespace` (1-dot) cross-namespace lookups.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: inject-ndots-2
spec:
  rules:
    - name: set-ndots-2
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: ["!kube-system"]
      mutate:
        patchStrategicMerge:
          spec:
            dnsConfig:
              options:
                - name: ndots
                  value: "2"
                - name: single-request-reopen
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (3).

ndots:2 still expands 0-dot (redis) and 1-dot (auth.payments) internal names via .svc.cluster.local, whereas ndots:1 sends auth.payments to root DNS first. force_tcp avoids UDP conntrack drops between nodes and CoreDNS.

ChoiceWhat it buysWhat it costs
Trade-off 1TCP upstream adds a tiny connection-setup cost amortized over persistent connections, while eliminating UDP packet loss.

Test `dots < ndots`

for ndots=2, 0 < 2 and 1 < 2 (search path used); 2 >= 2 (external root first).

Step 2

Use NOTRACK on 169.254.20.10 for pod-to-NodeLocal UDP.

Step 3

Use force_tcp from NodeLocal to CoreDNS so kube-proxy NAT never drops UDP.

Alternative 1

Use Cilium eBPF socket-level load balancing to remove iptables UDP conntrack races.

Using `ndots

1 causes auth.payments to leak internal namespace names to external DNS servers and return NXDOMAIN`.

def resolves_internally_first(name: str, ndots: int = 2) -> bool:
    # Names with fewer than ndots try .svc.cluster.local suffixes first
    return name.count(".") < ndots
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (4).

Enable serve_stale in CoreDNS/NodeLocal (cache 30 { serve_stale 5m immediate }) so transient upstream VPC DNS throttling serves slightly stale records instead of SERVFAIL.

ChoiceWhat it buysWhat it costs
Trade-off 1`serve_stale` prioritizes availability over strict TTL freshness during an upstream DNS outage.

Step 1

Add serve_stale 5m immediate to CoreDNS cache plugin.

Step 2

Configure max_concurrent and health / ready timeouts.

Step 3

Increase VPC resolver / Route 53 Resolver inbound endpoint capacity if throttled.

Alternative 1

Run redundant upstream resolvers (e.g., VPC DNS + Unbound fallback).

Failure Mode 1

Setting serve_stale for too long (e.g., 24h) can pin traffic to a decommissioned IP after an outage.

# Corefile cache block with serve_stale
.:53 {
    cache 30 {
        success 9984 30
        denial 9984 15
        serve_stale 5m immediate
    }
    forward . /etc/resolv.conf {
        max_concurrent 1000
    }
}
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Implementation & Guardrails (5).

Track coredns_dns_responses_total{rcode='NXDOMAIN'} / total and node_nf_conntrack_stat_drop, alerting if NXDOMAIN ratio exceeds 40% or p99 DNS latency exceeds 20 ms.

ChoiceWhat it buysWhat it costs
Trade-off 1Monitoring `NXDOMAIN` ratio catches misconfigured `ndots:5` workloads before they saturate CoreDNS.

Scrape CoreDNS `

9153 and NodeLocal :9253` metrics in Prometheus.

Step 2

Alert on NXDOMAIN ratio > 40% (indicates ndots:5 regression).

Step 3

Alert on coredns_dns_request_duration_seconds p99 > 20 ms.

Alternative 1

Use Hubble / eBPF DNS observability to attribute top NXDOMAIN callers by pod label.

Failure Mode 1

Averaging CoreDNS latency across all nodes hides single-node conntrack drops.

def check_dns_health(nxdomain_qps: float, total_qps: float, p99_ms: float) -> str:
    nx_ratio = nxdomain_qps / max(1.0, total_qps)
    if p99_ms > 50.0:
        return "CRITICAL_DNS_LATENCY"
    if nx_ratio > 0.40:
        return "WARN_NDOTS_AMPLIFICATION"
    return "HEALTHY"

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Deploy NodeLocal DNSCache (169.254.20.10 with TCP upstream), lower pod ndots to 2 (or use trailing dots api.stripe.com.), and tune CoreDNS cache + autopath.
  • 02Trade-off tip: TCP upstream adds a tiny connection-setup cost amortized over persistent connections, while eliminating UDP packet loss.
  • 03Watch out for: Setting serve_stale for too long (e.g., 24h) can pin traffic to a decommissioned IP after an outage.
Put it together · Synthesis

Putting Fixing CoreDNS Query Flooding from ndots:5 Together

Deploy `NodeLocal DNSCache` (`169.254.20.10` with TCP upstream), lower pod `ndots` to `2` (or use trailing dots `api.stripe.com.`), and tune CoreDNS `cache` + `autopath`.

A candidate architecture, not a universal answer

Putting Fixing CoreDNS Query Flooding from ndots:5 Together

Step 1
Step 2
Step 3
  1. Deploy node-local-dns DaemonSet on 169.254.20.10 and point kubelet --cluster-dns to it.
  2. Apply Kyverno mutation policy setting ndots: 2.
  3. Verify NXDOMAIN rate drops >80% in Prometheus.
Enable `serve_stale` in CoreDNS/NodeLocal (`cache 30 { serve_stale 5m immediate }`) so transient upstream VPC DNS throttling serves slightly stale records instead of `SERVFAIL`.
Interview Readiness Check

Enable serve_stale in CoreDNS/NodeLocal (cache 30 { serve_stale 5m immediate }) so transient upstream VPC DNS throttling serves slightly stale records instead of SERVFAIL.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Fixing CoreDNS Query Flooding from ndots:5

Deploy NodeLocal DNSCache (169.254.20.10 with TCP upstream), lower pod ndots to 2 (or use trailing dots api.stripe.com.), and tune CoreDNS cache + autopath.