Canary Deployment Pattern: Progressive Delivery Guarded by SLOs

Scenario context

Canary deployment is useful only when the rollout gates are defined before the rollout starts. Sending 1% of traffic to a bad build is still a bad release if you do not know what metric forces rollback.

sequenceDiagram
  participant C as Client
  participant R as Traffic Router
  participant S as Stable v1
  participant Ca as Canary v2
  participant M as Metrics Collector
  C->>R: HTTP request
  alt 95% of traffic
    R->>S: forward to stable
    S-->>C: response
  else 5% of traffic
    R->>Ca: forward to canary
    Ca-->>C: response
  end
  R->>M: record error rate and p99
  M-->>R: gate: pass or rollback signal

Observable outcome: 📊 Traffic Splitting: Router Sends 5% to Canary This sequence diagram shows how a traffic router implements the 95/5 split between the stable and canary versions of a service.

Stage 4 — Concrete Config Example: Argo Rollouts Canary

• This Argo Rollouts manifest implements progressive canary delivery for a search API, encoding traffic steps and SLO-based analysis gates directly into the Kubernetes rollout spec.
• This config directly implements the canary promotion decision tree shown in the diagrams above — each setWeight step increases canary traffic only after the preceding analysis run confirms that error rate and latency th…
• Read the steps array from top to bottom: every setWeight increases canary exposure, every pause holds for an observation window, and every analysis reference evaluates the SLO metrics before the rollout is allowed to co…
• Why operators care about this shape: - setWeight forces explicit exposure stages.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: search-api
spec:
  replicas: 12
  strategy:
    canary:
      stableService: search-api-stable
      canaryService: search-api-canary
      steps:
        - setWeight: 5
        - paus
e:
            duration: 10m
        - analysi
s:
            templates:
              - templateName: search-latency-and-errors
        - setWeight: 25
        - paus
e:
            duration: 15m
  selector:
    matchLabels:
      app: search-api
  template:
    metadata:
      labels:
        app: search-api
    spec:
      containers:
        - nam
e: api
          image: ghcr.io/abstractalgorithms/search-api:4.12.0
          ports:
            - containerPort: 8080

Observable outcome: Why operators care about this shape: - setWeight forces explicit exposure stages.

Stage 5 — Real-World Applications: What to Instrument and What to Compare

• Canary without comparison discipline becomes theatre.
• Compare candidate to stable at the same time window.
• Compare by cohort or region if traffic shapes differ.
• Use absolute thresholds and relative deltas.

Observable outcome: Use absolute thresholds and relative deltas.

Stage 6 — Spring Boot Health Endpoint: SLO-Based Traffic Gate for Canary Promotion

• Spring Boot Actuator's /actuator/health endpoint is the standard HTTP target for canary promotion gates in Argo Rollouts, Flagger, and Istio.
• By composing custom HealthIndicator beans that evaluate the SLO signals described in the Canary Gate Checklist above — error rate, p99 latency, business proxy — a Spring Boot service self-reports whether promotion shoul…
• How it solves the problem: The AnalysisTemplate in Argo Rollouts and the Canary metric spec in Flagger both need an HTTP endpoint that returns a machine-readable pass/fail result.
• A Spring Boot HealthIndicator that reads Micrometer counters and timers provides exactly that — promotion gates become measurable code rather than human judgment.

// SLO-based canary health indicator — evaluated by Argo analysis probe
@Component("canaryReadinessGate")
public class CanaryPromotionHealthIndicator implements HealthIndicator {

    private final MeterRegistry registry;

    // Thresholds defined before rollout — not after the fact
    private static final double MAX_ERROR_RATE        = 0.005;  // 0.5% error budget
    private static final double MAX_P99_LATENCY_MS    = 250.0;
    private static final double MIN_CHECKOUT_SUCCESS  = 0.98;   // business KPI proxy

    public CanaryPromotionHealthIndicator(MeterRegistry registry) {
        this.registry = registry;
    }

    @Override
    public Health health() {
        Map<String, Object> details = new LinkedHashMap<>();

        // SLI 1: request error rate (via Micrometer counter tags)
        double totalRequests = registry.get("http.server.requests").timer().count();
        double errorRequests = registry.get("http.server.requests")
            .tag("status", "5xx").timer().count();
        double errorRate = totalRequests > 0 ? errorRequests / totalRequests : 0.0;
        details.put("errorRate", String.format("%.4f", errorRate));

        if (errorRate > MAX_ERROR_RATE) {
            return Health.down().withDetails(details)
                .withDetail("reason", "error rate exceeds SLO threshold").build();
        }

        // SLI 2: p99 latency from Micrometer timer percentile
        double p99Ms = registry.get("http.server.requests")
            .timer().percentile(0.99) / 1_000_000.0;  // nanoseconds → ms
        details.put("p99LatencyMs", String.format("%.1f", p99Ms));

        if (p99Ms > MAX_P99_LATENCY_MS) {
            return Health.down().withDetails(details)
                .withDetail("reason", "p99 latency exceeds promotion threshold").build();
        }

        // SLI 3: business proxy — checkout success rate
        double checkoutAttempts = registry.get("checkout.attempts").counter().count();
        double checkoutSuccess  = registry.get("checkout.success").counter().count();
        double checkoutRate = checkoutAttempts > 0
            ? checkoutSuccess / checkoutAttempts : 1.0;
        details.put("checkoutSuccessRate", String.format("%.4f", checkoutRate));

        if (checkoutRate < MIN_CHECKOUT_SUCCESS) {
            return Health.down().withDetails(details)
                .withDetail("reason", "checkout success rate below business threshold").build();
        }

        return Health.up().withDetails(details)
            .withDetail("promotionGate", "PASS").build();
    }
}

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: search-latency-and-errors
spec:
  metrics:
    - nam
e: slo-health-gate
      interval: 15s
      successCondition: "result == 'UP'"
      failureLimit: 2
      provider:
        web:
          url: http://search-api-canary/actuator/health/canaryReadinessGate
          jsonPath: "{$.status}"

Observable outcome: A Spring Boot HealthIndicator that reads Micrometer counters and timers provides exactly that — promotion gates become measurable code rather than human judgment.

End-to-end outcome

Canary is a progressive delivery pattern, not just a traffic-splitting feature.

Alternatives considered

Decision Guide for SRE Teams

If you cannot answer “what exact metric trips rollback at 5% traffic?”, the service is not canary ready.

Failure and degraded modes

When Not to Use Canary

• Detection: Canary is weak when the sample is too small to be meaningful or when the real danger is schema irreversibility rather than serving-path behavior.
• User or operator impact: Canary is weak when the sample is too small to be meaningful or when the real danger is schema irreversibility rather than serving-path behavior.

Trade-offs & Failure Modes: Pros, Cons, and Alternatives

• Detection: Trade-offs & Failure Modes: Pros, Cons, and Alternatives
• User or operator impact: Trade-offs & Failure Modes: Pros, Cons, and Alternatives