An e-commerce checkout service deploys a 5% canary (v2.8) behind an AWS ALB configured with target-group sticky sessions (`stickiness.enabled=true`, 24-hour cookie `AWSALB`). Automated canary analysis (Kayenta/Argo Rollouts) passes v2.8 after 30 minutes because error rate is 0.01% and p99 is 42 ms. Upon promoting v2.8 to 100%, authenticated checkout fails at 14% with a `NullPointerException` on saved payment tokens. Investigation reveals v2.8 only received 0.4% of traffic—exclusively new anonymous visitors—while 99.6% of returning authenticated users were pinned to Baseline by the 24-hour sticky cookie. Design a bias-free canary routing and statistical evaluation pipeline.
From patterns to decisions.
An e-commerce checkout service deploys a 5% canary (v2.8) behind an AWS ALB configured with target-group sticky sessions (`stickiness.enabled=true`, 24-hour cookie `AWSALB`). Automated canary analysis (Kayenta/Argo Rollouts) passes v2.8 after 30 minutes because error rate is 0.01% and p99 is 42 ms. Upon promoting v2.8 to 100%, authenticated checkout fails at 14% with a `NullPointerException` on saved payment tokens. Investigation reveals v2.8 only received 0.4% of traffic—exclusively new anonymous visitors—while 99.6% of returning authenticated users were pinned to Baseline by the 24-hour sticky cookie. Design a bias-free canary routing and statistical evaluation pipeline.
The 24-hour AWSALB sticky cookie.
The 24-hour AWSALB sticky cookie pinned 85% of returning authenticated users to Baseline, starving Canary of the exact cohort that exercises saved payment tokens.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Sticky sessions simplify legacy stateful apps but invalidate A/B and canary statistics |
Step 1
Trace how AWSALB cookies bypass weighted target-group selection on returning users.
Compute effective canary share
15% anonymous x 5% weight = 0.75% max (0.4% observed).
Step 3
Explain why automated canary analysis passed on a non-representative sample.
Alternative 1
Shorten cookie TTL (still biases returning users who stay active)
Failure Mode 1
False-positive canary promotion leading to a 100% production outage
Expected Canary Auth RPS: 3,400 * 0.05 = 170 RPS
Actual Canary Auth RPS: ~0 RPS (pinned to Baseline by 24h AWSALB cookie)
Move sessions to Redis, disable.
Move sessions to Redis, disable ALB stickiness, and gate promotion on verified auth=true traffic share (4–6%) and error-rate parity.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Requires instrumenting `auth="true|false"` as a low-cardinality Prometheus label on HTTP metrics |
Step 1
Migrate session state to Redis/JWTs and set stickiness.enabled=false.
Step 2
Route 5% per-request (or by hash(x-user-id)).
Step 3
Add canary-auth-traffic-share and canary-auth-error-rate checks to AnalysisTemplate.
Alternative 1
Use Kayenta with separate metric scopes for authenticated and anonymous cohorts
Failure Mode 1
Using high-cardinality user_id labels in Prometheus instead of boolean auth cohorts crashes TSDB
metrics:
- name: canary-auth-traffic-share
successCondition: result[0] >= 0.04 && result[0] <= 0.06
query: |
sum(rate(http_requests_total{version="canary",auth="true"}[5m])) /
sum(rate(http_requests_total{auth="true"}[5m]))
Use deterministic consistent hashing on.
Use deterministic consistent hashing on user_id (or session_id) at the L7 proxy so 5% of all users map to Canary regardless of prior visits.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Requires an edge/API-gateway layer to compute the hash or inject `x-canary-cohort` before ALB/Istio |
Step 1
Hash X-User-ID (e.g., MurmurHash(user_id) % 100 < 5).
Step 2
Unlike AWSALB cookies, this is stateless and newly applied on every release.
Step 3
Every cohort (new and returning) is split at an exact 5% ratio.
Alternative 1
Make checkout v2.7 and v2.8 API- and state-compatible so per-request random splitting is completely seamless
Failure Mode 1
Using the same static hash seed across all releases pins the same 5% of users to every canary permanently
http:
- match:
- headers:
x-canary-cohort:
exact: "true"
route:
- destination: { host: checkout, subset: canary }
Implementation & Guardrails (4).
Add an explicit minimum-sample-size (Inconclusive) check in the AnalysisTemplate so promotion pauses until N >= 5,000 requests.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Delays overnight promotions until morning traffic ramps up, trading speed for safety |
Step 1
Query sum(increase(http_requests_total{version="canary",auth="true"}[30m])).
Step 2
Require >= 5000 before successCondition can evaluate to true.
Step 3
If under 5,000, extend the canary window or require manual SRE sign-off.
Alternative 1
Run synthetic canary traffic generators exercising saved-card checkout paths in production
Failure Mode 1
Dividing by zero or near-zero RPS in error-rate queries without clamp_min(..., 1)
- name: min-canary-sample-size
successCondition: result[0] >= 5000
inconclusiveLimit: 10
query: sum(increase(http_requests_total{version="canary",auth="true"}[30m]))
Use Envoy/Istio traffic shadowing (mirror).
Use Envoy/Istio traffic shadowing (mirror) for read/idempotent endpoints, dropping mirrored responses while logging v2.8 exceptions and latency.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Shadowing doubles backend read load for the mirrored percentage and requires idempotent/no-op write guards |
Step 1
Configure Envoy request_mirror_policies at 5–10% to checkout-canary.
Tag mirrored requests with `X-Shadow-Request
true` so v2.8 skips payment-gateway charges and DB writes.
Step 3
Inspect v2.8 logs and metrics for NullPointerExceptions with zero user impact.
Alternative 1
Replay sanitized production request logs in a staging environment against v2.8
Failure Mode 1
Forgetting to disable side effects (emails, Stripe charges) on shadowed requests causes duplicate charges
mirror:
host: checkout-canary.prod.svc.cluster.local
mirrorPercentage:
value: 10.0
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Eliminate session-affinity skew by using stateless services with JWT/Redis sessions, request-level weighted routing, and stratified canary evaluation.
- 02Trade-off tip: Requires an edge/API-gateway layer to compute the hash or inject
x-canary-cohortbefore ALB/Istio - 03Watch out for: Dividing by zero or near-zero RPS in error-rate queries without
clamp_min(..., 1)
Unbiased Canary Deployment Evaluation — Normal → Load / Fault → Recovery
Eliminate session-affinity skew by using stateless services with JWT/Redis sessions, request-level weighted routing, and stratified canary evaluation.
Putting Unbiased Canary Deployments & Metric Evaluation Together
Eliminate session-affinity skew by using stateless services with JWT/Redis sessions, request-level weighted routing, and stratified canary evaluation.
Putting Unbiased Canary Deployments & Metric Evaluation Together
- Migrate session state to Redis/JWTs and set
stickiness.enabled=false. - Route 5% per-request (or by
hash(x-user-id)). - Add
canary-auth-traffic-shareandcanary-auth-error-ratechecks toAnalysisTemplate.
Interview Readiness Check
Add an explicit minimum-sample-size (Inconclusive) check in the AnalysisTemplate so promotion pauses until N >= 5,000 requests.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Unbiased Canary Deployments & Metric Evaluation
Eliminate session-affinity skew by using stateless services with JWT/Redis sessions, request-level weighted routing, and stratified canary evaluation.
