Deployment Architecture Patterns: Blue-Green, Canary, Shadow Traffic, Feature Flags, and GitOps

Scenario context

Release safety is an architecture capability, not just a CI/CD convenience. Blue-green, canary, shadow traffic, feature flags, and GitOps patterns exist to control blast radius, measure regressions early, and make rollback fast enough to matter.

TLDR: Safe deployments are controlled experiments: limit exposure, measure quickly, and make rollback boring.

🚨 The Problem This Solves

In 2021, a fintech rele…

This walkthrough focuses on Why Deployment Patterns Belong in Architecture Reviews.

Before making a design choice, clarify:

• Deployment design determines failure blast radius just as much as service design.
• If rollout controls are weak, good code still creates bad incidents.
• Practical review questions: - How fast can we detect regression?

Concepts in play: canary-deployments, caching, git-internals-and-versioning, api-gateway

Stage 1 — When to Use Blue-Green, Canary, Shadow, Flags, and GitOps

• When not to overcomplicate - If service changes are low-risk and rare, basic canary may be enough.
• • If you cannot measure business impact, progressive rollout gives false confidence.
• 📊 Deployment Pipeline States This state diagram captures the distinct lifecycle states for two deployment strategies side by side.
• BlueGreen moves through GreenLive → BlueDeploy → BlueValidate → BlueLive in a hard cutover, while Canary progresses incrementally through small, broad, and full traffic slices before completing.

stateDiagram-v2
    state BlueGreen {
        [*] --> GreenLive
        GreenLive --> BlueDeploy
        BlueDeploy --> BlueValidate
        BlueValidate --> BlueLive
        BlueLive --> [*]
    }
    state Canary {
        [*] --> CanarySmall
        CanarySmall --> CanaryBroad
        CanaryBroad --> CanaryFull
        CanaryFull --> [*]
    }

Observable outcome: BlueGreen moves through GreenLive → BlueDeploy → BlueValidate → BlueLive in a hard cutover, while Canary progresses incrementally through small, broad, and full traffic slices before completing.

Stage 2 — How the Release Control Loop Works

• Promote artifact to release candidate.
• Deploy through declarative desired state (GitOps or equivalent).
• Start canary slice and evaluate technical + business signals.
• Flip feature flags per cohort if needed.

Observable outcome: Flip feature flags per cohort if needed.

Stage 3 — How to Implement: Progressive Delivery Checklist

• Define rollout gates (error, latency, saturation, business KPI).
• Define stop conditions and automatic rollback thresholds.
• Add traffic-routing primitives (weights or ring cohorts).
• Separate deploy from expose with feature flags.

Observable outcome: Separate deploy from expose with feature flags.

Stage 4 — Deep Dive: Stateful Releases, Signal Quality, and Rollback Reality

• The Internals: Desired State + Runtime Gates GitOps controls desired state, but runtime safety still depends on gates and reversible data changes.
• Keep these concerns separate: - deployment : where code is running, - traffic : how much real traffic it receives, - feature exposure : which users see new behavior, - data compatibility : whether old and new versions c…
• Stateful change rule: never require immediate irreversible data transformation to keep serving.
• Performance Analysis: Metrics That Matter Most

Observable outcome: Performance Analysis: Metrics That Matter Most

Stage 5 — Rollout Flow: Deploy, Observe, Expand, or Revert

• This diagram shows the end-to-end progressive delivery loop from CI artifact through GitOps desired-state commit, controller deployment, shadow checks, and staged canary traffic expansion.
• The gate check after initial canary exposure is the critical decision point: a passing gate expands ring by ring until feature flags complete exposure, while a failing gate triggers an immediate rollback of both traffic…
• The key takeaway is that safety comes from gating every expansion step, not from deploying slowly.
• 📊 Traffic Routing Comparison This diagram contrasts how traffic is split across BlueGreen, Canary, and Shadow strategies at the load balancer level.

flowchart TD
  A[CI artifact] --> B[GitOps desired state commit]
  B --> C[Controller deploys candidate]
  C --> D[Shadow checks and smoke tests]
  D --> E[Canary 1-5 percent traffic]
  E --> F{Gates pass?}
  F -->|Yes| G[Expand traffic ring by ring]
  G --> H[Enable feature flags by cohort]
  F -->|No| I[Rollback traffic and release]

flowchart LR
    subgraph BlueGreen
        LB1[Load Balancer] -->|100%| BG1[Blue v1]
        LB1 -.->|0% cutover| BG2[Green v2]
    end
    subgraph Canary
        LB2[Load Balancer] -->|95%| C1[Stable v1]
        LB2 -->|5%| C2[Canary v2]
    end
    subgraph Shadow
        LB3[Load Balancer] -->|100%| S1[Live v1]
        LB3 -.->|mirror| S2[Shadow v2]
    end

Observable outcome: 📊 Traffic Routing Comparison This diagram contrasts how traffic is split across BlueGreen, Canary, and Shadow strategies at the load balancer level.

Stage 6 — Practical Example: Canary Policy With Automatic Abort

• The safest rollout controllers encode traffic steps and abort conditions directly in config so the happy path and the rollback path use the same source of truth.
• Operational checks that matter more than the syntax: 1.
• The pause window has to be longer than the metric stabilization window, or the gate is decorative.
• Technical and business guardrails should both participate in abort decisions.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: recommendation-api
spec:
  replicas: 12
  strategy:
    canary:
      maxUnavailable: 0
      canaryService: recommendation-api-canary
      stableService: recommendation-api-stable
      steps:
        - setWeight: 5
        - paus
e:
            duration: 10m
        - analysi
s:
            templates:
              - templateName: canary-errors
              - templateName: conversion-guardrail
        - setWeight: 25
        - paus
e:
            duration: 20m

Observable outcome: Technical and business guardrails should both participate in abort decisions.

End-to-end outcome

After working through the stages, you should be able to explain:

• • Deploy and expose are different control planes and should stay separate.
• • Canary and shadow only work with representative traffic and meaningful gates.

Alternatives considered

Decision Guide: Picking a Rollout Pattern Fast

Use combinations deliberately, not by default.

Failure and degraded modes

Trade-offs & Failure Modes: Pros, Cons, and Risks

• Detection: Trade-offs & Failure Modes: Pros, Cons, and Risks
• User or operator impact: Trade-offs & Failure Modes: Pros, Cons, and Risks