Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Unbiased Canary Deployments
& Metric Evaluation.

An e-commerce checkout service deploys a 5% canary (v2.8) behind an AWS ALB configured with target-group sticky sessions (stickiness.enabled=true, 24-hour cookie AWSALB). Automated canary analysis (Kayenta/Argo Rollouts) passes v2.8 after 30 minutes because error rate is 0.01% and p99 is 42 ms. Upon promoting v2.8 to 100%, authenticated checkout fails at 14% with a NullPointerException on saved payment tokens. Investigation reveals v2.8 only received 0.4% of traffic—exclusively new anonymous visitors—while 99.6% of returning authenticated users were pinned to Baseline by the 24-hour sticky cookie. Design a bias-free canary routing and statistical evaluation pipeline.
Start the journey5 stages
Loading diagram…
Unbiased Canary Deployments & Metric Evaluation architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

An e-commerce checkout service deploys a 5% canary (v2.8) behind an AWS ALB configured with target-group sticky sessions (`stickiness.enabled=true`, 24-hour cookie `AWSALB`). Automated canary analysis (Kayenta/Argo Rollouts) passes v2.8 after 30 minutes because error rate is 0.01% and p99 is 42 ms. Upon promoting v2.8 to 100%, authenticated checkout fails at 14% with a `NullPointerException` on saved payment tokens. Investigation reveals v2.8 only received 0.4% of traffic—exclusively new anonymous visitors—while 99.6% of returning authenticated users were pinned to Baseline by the 24-hour sticky cookie. Design a bias-free canary routing and statistical evaluation pipeline.

From patterns to decisions.

An e-commerce checkout service deploys a 5% canary (v2.8) behind an AWS ALB configured with target-group sticky sessions (`stickiness.enabled=true`, 24-hour cookie `AWSALB`). Automated canary analysis (Kayenta/Argo Rollouts) passes v2.8 after 30 minutes because error rate is 0.01% and p99 is 42 ms. Upon promoting v2.8 to 100%, authenticated checkout fails at 14% with a `NullPointerException` on saved payment tokens. Investigation reveals v2.8 only received 0.4% of traffic—exclusively new anonymous visitors—while 99.6% of returning authenticated users were pinned to Baseline by the 24-hour sticky cookie. Design a bias-free canary routing and statistical evaluation pipeline.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

The 24-hour AWSALB sticky cookie.

The 24-hour AWSALB sticky cookie pinned 85% of returning authenticated users to Baseline, starving Canary of the exact cohort that exercises saved payment tokens.

ChoiceWhat it buysWhat it costs
Trade-off 1Sticky sessions simplify legacy stateful apps but invalidate A/B and canary statistics

Step 1

Trace how AWSALB cookies bypass weighted target-group selection on returning users.

Compute effective canary share

15% anonymous x 5% weight = 0.75% max (0.4% observed).

Step 3

Explain why automated canary analysis passed on a non-representative sample.

Alternative 1

Shorten cookie TTL (still biases returning users who stay active)

Failure Mode 1

False-positive canary promotion leading to a 100% production outage

Expected Canary Auth RPS: 3,400 * 0.05 = 170 RPS
Actual Canary Auth RPS: ~0 RPS (pinned to Baseline by 24h AWSALB cookie)
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Move sessions to Redis, disable.

Move sessions to Redis, disable ALB stickiness, and gate promotion on verified auth=true traffic share (4–6%) and error-rate parity.

ChoiceWhat it buysWhat it costs
Trade-off 1Requires instrumenting `auth="true|false"` as a low-cardinality Prometheus label on HTTP metrics

Step 1

Migrate session state to Redis/JWTs and set stickiness.enabled=false.

Step 2

Route 5% per-request (or by hash(x-user-id)).

Step 3

Add canary-auth-traffic-share and canary-auth-error-rate checks to AnalysisTemplate.

Alternative 1

Use Kayenta with separate metric scopes for authenticated and anonymous cohorts

Failure Mode 1

Using high-cardinality user_id labels in Prometheus instead of boolean auth cohorts crashes TSDB

metrics:
  - name: canary-auth-traffic-share
    successCondition: result[0] >= 0.04 && result[0] <= 0.06
    query: |
      sum(rate(http_requests_total{version="canary",auth="true"}[5m])) /
      sum(rate(http_requests_total{auth="true"}[5m]))
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

Use deterministic consistent hashing on.

Use deterministic consistent hashing on user_id (or session_id) at the L7 proxy so 5% of all users map to Canary regardless of prior visits.

ChoiceWhat it buysWhat it costs
Trade-off 1Requires an edge/API-gateway layer to compute the hash or inject `x-canary-cohort` before ALB/Istio

Step 1

Hash X-User-ID (e.g., MurmurHash(user_id) % 100 < 5).

Step 2

Unlike AWSALB cookies, this is stateless and newly applied on every release.

Step 3

Every cohort (new and returning) is split at an exact 5% ratio.

Alternative 1

Make checkout v2.7 and v2.8 API- and state-compatible so per-request random splitting is completely seamless

Failure Mode 1

Using the same static hash seed across all releases pins the same 5% of users to every canary permanently

http:
  - match:
      - headers:
          x-canary-cohort:
            exact: "true"
    route:
      - destination: { host: checkout, subset: canary }
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (4).

Add an explicit minimum-sample-size (Inconclusive) check in the AnalysisTemplate so promotion pauses until N >= 5,000 requests.

ChoiceWhat it buysWhat it costs
Trade-off 1Delays overnight promotions until morning traffic ramps up, trading speed for safety

Step 1

Query sum(increase(http_requests_total{version="canary",auth="true"}[30m])).

Step 2

Require >= 5000 before successCondition can evaluate to true.

Step 3

If under 5,000, extend the canary window or require manual SRE sign-off.

Alternative 1

Run synthetic canary traffic generators exercising saved-card checkout paths in production

Failure Mode 1

Dividing by zero or near-zero RPS in error-rate queries without clamp_min(..., 1)

- name: min-canary-sample-size
  successCondition: result[0] >= 5000
  inconclusiveLimit: 10
  query: sum(increase(http_requests_total{version="canary",auth="true"}[30m]))
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Use Envoy/Istio traffic shadowing (mirror).

Use Envoy/Istio traffic shadowing (mirror) for read/idempotent endpoints, dropping mirrored responses while logging v2.8 exceptions and latency.

ChoiceWhat it buysWhat it costs
Trade-off 1Shadowing doubles backend read load for the mirrored percentage and requires idempotent/no-op write guards

Step 1

Configure Envoy request_mirror_policies at 5–10% to checkout-canary.

Tag mirrored requests with `X-Shadow-Request

true` so v2.8 skips payment-gateway charges and DB writes.

Step 3

Inspect v2.8 logs and metrics for NullPointerExceptions with zero user impact.

Alternative 1

Replay sanitized production request logs in a staging environment against v2.8

Failure Mode 1

Forgetting to disable side effects (emails, Stripe charges) on shadowed requests causes duplicate charges

mirror:
  host: checkout-canary.prod.svc.cluster.local
mirrorPercentage:
  value: 10.0

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Eliminate session-affinity skew by using stateless services with JWT/Redis sessions, request-level weighted routing, and stratified canary evaluation.
  • 02Trade-off tip: Requires an edge/API-gateway layer to compute the hash or inject x-canary-cohort before ALB/Istio
  • 03Watch out for: Dividing by zero or near-zero RPS in error-rate queries without clamp_min(..., 1)
Behavioral Progression · Unbiased Canary Deployment Evaluation

Unbiased Canary Deployment Evaluation — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Eliminate session-affinity skew by using stateless services with JWT/Redis sessions, request-level weighted routing, and stratified canary evaluation.

Synthesis / One connected design

Putting Unbiased Canary Deployments & Metric Evaluation Together

Eliminate session-affinity skew by using stateless services with JWT/Redis sessions, request-level weighted routing, and stratified canary evaluation.

A candidate architecture, not a universal answer

Putting Unbiased Canary Deployments & Metric Evaluation Together

Step 1
Step 2
Step 3
  1. Migrate session state to Redis/JWTs and set stickiness.enabled=false.
  2. Route 5% per-request (or by hash(x-user-id)).
  3. Add canary-auth-traffic-share and canary-auth-error-rate checks to AnalysisTemplate.
Add an explicit minimum-sample-size (`Inconclusive`) check in the `AnalysisTemplate` so promotion pauses until `N >= 5,000` requests.
Interview Readiness Check

Add an explicit minimum-sample-size (Inconclusive) check in the AnalysisTemplate so promotion pauses until N >= 5,000 requests.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Unbiased Canary Deployments & Metric Evaluation

Eliminate session-affinity skew by using stateless services with JWT/Redis sessions, request-level weighted routing, and stratified canary evaluation.