Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Eliminating Alert Fatigue with
SLO Burn-Rate Alerts.

Your payments platform processes 12,000 RPS across 180 Kubernetes pods. Over the past quarter, on-call SREs averaged 420 PagerDuty pages per week (60/day), 88% of which self-resolved within 3 minutes without customer impact. Last Tuesday, a real 22-minute P1 database connection-pool exhaustion in checkout-service was buried under 47 downstream timeout pages and auto-acknowledged by an exhausted engineer. Redesign the alerting and paging pipeline so only actionable, user-impacting incidents wake on-call engineers while preserving observability for transient anomalies.
Start the journey5 stages
Loading diagram…
Eliminating Alert Fatigue with SLO Burn-Rate Alerts architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your payments platform processes 12,000 RPS across 180 Kubernetes pods. Over the past quarter, on-call SREs averaged 420 PagerDuty pages per week (60/day), 88% of which self-resolved within 3 minutes without customer impact. Last Tuesday, a real 22-minute P1 database connection-pool exhaustion in checkout-service was buried under 47 downstream timeout pages and auto-acknowledged by an exhausted engineer. Redesign the alerting and paging pipeline so only actionable, user-impacting incidents wake on-call engineers while preserving observability for transient anomalies.

From patterns to decisions.

Your payments platform processes 12,000 RPS across 180 Kubernetes pods. Over the past quarter, on-call SREs averaged 420 PagerDuty pages per week (60/day), 88% of which self-resolved within 3 minutes without customer impact. Last Tuesday, a real 22-minute P1 database connection-pool exhaustion in checkout-service was buried under 47 downstream timeout pages and auto-acknowledged by an exhausted engineer. Redesign the alerting and paging pipeline so only actionable, user-impacting incidents wake on-call engineers while preserving observability for transient anomalies.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

Symptomless threshold alerts (CPU >85%.

Symptomless threshold alerts (CPU >85%, 1-minute 5xx spikes) created chronic false positives, while lack of dependency inhibition turned a single DB pool outage into 47 simultaneous pages.

ChoiceWhat it buysWhat it costs
Trade-off 1Auditing historical incidents delays rule changes by 1-2 days but prevents accidentally silencing genuine P1 signals.

Step 1

Audit PagerDuty incident logs over the last 30 days to compute the auto-resolve percentage per alert rule.

Step 2

Map upstream-to-downstream service dependencies to identify cascading alert storms.

Step 3

Define user-facing SLIs (HTTP 5xx ratio and p99 latency) and monthly error budgets.

Alternative 1

Immediate blanket suppression of all non-5xx alerts (risky without baseline SLO coverage).

Failure Mode 1

Confusing short-duration pages caused by human quick-fixes with genuine self-healing false positives.

-- Audit noisy alert rules from exported PagerDuty incident telemetry
SELECT alert_name,
       COUNT(*) AS total_pages,
       ROUND(100.0 * SUM(CASE WHEN duration_sec < 180 THEN 1 ELSE 0 END) / COUNT(*), 1) AS pct_auto_resolved
FROM pagerduty_incidents
WHERE created_at >= NOW() - INTERVAL '30 days'
GROUP BY alert_name
HAVING COUNT(*) > 10
ORDER BY pct_auto_resolved DESC;
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Combine 14.

Combine 14.4x (1h & 5m) and 6x (6h & 30m) burn-rate rules with minimum RPS guards, Alertmanager inhibit_rules, and deterministic PagerDuty dedup_key templates.

ChoiceWhat it buysWhat it costs
Trade-off 1Multi-window rules require 5-60 minutes of metric history; new services need 1 hour of warmup before burn-rate alerts stabilize.

Step 1

Create Prometheus recording rules for 5m, 30m, 1h, and 6h error ratios.

Step 2

Gate paging alerts on both long and short windows exceeding the burn-rate threshold AND traffic > 10 RPS.

Step 3

Configure Alertmanager group_wait (30s), inhibit_rules for DB/upstream outages, and dedup_key without replica labels.

Alternative 1

Datadog or Sloth automated SLO generators that compile multi-window PromQL from a simple YAML SLO spec.

Failure Mode 1

Forgetting to strip the Prometheus 'replica' external label causes HA pairs to generate distinct dedup_keys.

inhibit_rules:
  - source_matchers: ['alertname="PostgresPrimaryDown"', 'severity="page"']
    target_matchers: ['severity=~"page|warning"', 'upstream_dep="postgres-primary"']
    equal: ['cluster']

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - dedup_key: '{{ .GroupLabels.cluster }}:{{ .GroupLabels.service }}:{{ .GroupLabels.alertname }}'
        severity: 'critical'
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (3).

Dual-window gating requires sustained error budget consumption while resetting immediately when the 5m window recovers, eliminating both micro-bursts and long reset tails.

ChoiceWhat it buysWhat it costs
Trade-off 1Gating on >10 RPS prevents false pages on admin endpoints, at the cost of requiring separate blackbox uptime checks for low-QPS APIs.

Step 1

Compare 2-minute static threshold (fires on any 2-minute deploy blip) vs. 14.4x 1h+5m dual window.

Step 2

Add minimum RPS floor (>10 RPS) for ratio alerts or route low-QPS services to synthetic blackbox probes.

Alternative 1

Bayesian anomaly detection (high operational complexity and opaque thresholds).

Failure Mode 1

Using a 1h window alone without a 5m window causes alerts to stay firing for 55 minutes after an incident is already resolved.

def should_page_burn_rate(errors_5m: int, total_5m: int, errors_1h: int, total_1h: int, slo_target: float = 0.9995) -> bool:
    if total_5m < 3000:  # < 10 RPS over 5m (300s)
        return False
    budget = 1.0 - slo_target
    burn_5m = (errors_5m / total_5m) / budget
    burn_1h = (errors_1h / max(total_1h, 1)) / budget
    return burn_5m >= 14.4 and burn_1h >= 14.4
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

A single-instance Alertmanager is a.

A single-instance Alertmanager is a single point of failure for paging; a 3-node gossip mesh plus an external Watchdog heartbeat guarantees delivery.

ChoiceWhat it buysWhat it costs
Trade-off 1Gossip mesh prevents duplicate pages under normal operation, but during a full gossip partition Alertmanager fails open (may send 2 pages rather than 0).

Step 1

Deploy 3 Alertmanager replicas across 3 AZs joined via --cluster.peer.

Step 2

Configure Prometheus to send alerts to ALL Alertmanager endpoints (not behind a single load balancer).

Step 3

Route an always-firing Watchdog alert to an external heartbeat monitor that pages if silenced for >5 minutes.

Alternative 1

Managed alerting SaaS (Grafana Cloud / PagerDuty AIOps) to offload Alertmanager cluster ops.

Failure Mode 1

Putting Alertmanager behind a single Kubernetes ClusterIP Service instead of headless DNS causes Prometheus to only reach 1 pod.

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager-0.monitoring:9093
            - alertmanager-1.monitoring:9093
            - alertmanager-2.monitoring:9093
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Implementation & Guardrails (5).

Continuous alert governance prevents threshold drift by flagging rules with high auto-resolve rates or zero-action acknowledgments every Monday.

ChoiceWhat it buysWhat it costs
Trade-off 1Automated weekly reports create mild process overhead but prevent alert rot over quarters.

Step 1

Query PagerDuty /incidents API for the prior 7 days.

Step 2

Compute auto-resolve rate and median time-to-resolve per alert service/title.

Step 3

Post a weekly Slack digest to service owners for any rule exceeding 20% noise.

Alternative 1

Enforcing CI/CD policy checks (Conftest/OPA) blocking any new PrometheusRule without a runbook_url and for-duration.

Failure Mode 1

Ignoring warning-tier Slack noise until engineers mute the #ops-triage channel too.

import requests

def audit_noisy_alerts(pd_token: str, since_iso: str) -> list[dict]:
    resp = requests.get(
        "https://api.pagerduty.com/incidents",
        headers={"Authorization": f"Token token={pd_token}"},
        params={"since": since_iso, "limit": 100, "statuses[]": "resolved"},
        timeout=10,
    )
    resp.raise_for_status()
    return [i for i in resp.json()["incidents"] if i.get("resolve_reason") is None]

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Replace symptom-based infrastructure thresholds with multi-window SLO burn-rate alerts, Alertmanager inhibition/grouping, and deterministic dedup_key routing.
  • 02Trade-off tip: Gating on >10 RPS prevents false pages on admin endpoints, at the cost of requiring separate blackbox uptime checks for low-QPS APIs.
  • 03Watch out for: Putting Alertmanager behind a single Kubernetes ClusterIP Service instead of headless DNS causes Prometheus to only reach 1 pod.
Behavioral Progression · Slo Based Alerting Burn Rates

Slo Based Alerting Burn Rates — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Replace symptom-based infrastructure thresholds with multi-window SLO burn-rate alerts, Alertmanager inhibition/grouping, and deterministic dedupkey routing.

Synthesis / One connected design

Putting Eliminating Alert Fatigue with SLO Burn-Rate Alerts Together

Replace symptom-based infrastructure thresholds with multi-window SLO burn-rate alerts, Alertmanager inhibition/grouping, and deterministic dedup_key routing.

A candidate architecture, not a universal answer

Putting Eliminating Alert Fatigue with SLO Burn-Rate Alerts Together

Step 1
Step 2
Step 3
  1. Create Prometheus recording rules for 5m, 30m, 1h, and 6h error ratios.
  2. Gate paging alerts on both long and short windows exceeding the burn-rate threshold AND traffic > 10 RPS.
  3. Configure Alertmanager group_wait (30s), inhibit_rules for DB/upstream outages, and dedup_key without replica labels.
A single-instance Alertmanager is a single point of failure for paging; a 3-node gossip mesh plus an external Watchdog heartbeat guarantees delivery.
Interview Readiness Check

A single-instance Alertmanager is a single point of failure for paging; a 3-node gossip mesh plus an external Watchdog heartbeat guarantees delivery.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Eliminating Alert Fatigue with SLO Burn-Rate Alerts

Replace symptom-based infrastructure thresholds with multi-window SLO burn-rate alerts, Alertmanager inhibition/grouping, and deterministic dedup_key routing.