Advanced Data Structures & Algorithms

Java Backend Modernization

Beginner's Guide to Data Structures

Agentic AI Systems

Data Engineering & Spark Mastery

Low-Level Design Interview Prep

Microservices Interview Roadmap

Design Patterns Interview Roadmap

Distributed Systems Interview Roadmap

Mastering Python Development

System Design Interview Prep

Distributed Systems Refresher

AI Engineering Launchpad

Architecture Patterns for Scalable Systems

RAG & Vector Search Mastery

Production DevOps & Reliability

Enterprise Patterns Playbook

Master Coding Interviews

Coding Techniques Interview Roadmap

Data Structures Interview Roadmap

Architecture Patterns Interview Roadmap

software-engineering-principles

Migrate your Spring Boot services from thread-pool exhaustion to JDK 21 virtual threads — the practical Spring developer guide

Adapting to Virtual Threads for Spring Developers

how-it-works-internals-explained

From WAL and MVCC to fsync and isolation levels — the internals behind every ACID guarantee in PostgreSQL, MySQL, Oracle, and SQL Server.

ACID Properties Explained: How SQL Databases Guarantee Atomicity, Consistency, Isolation, and Durability

How modern NoSQL databases achieve atomicity and isolation without a single machine

ACID Transactions in Distributed Databases: DynamoDB, Cosmos DB, and Spanner Compared

llm-engineering

Is an LLM a brain in a jar? To make it truly useful, we need to give it access to the world. This guide explains RAG and Agents.

Advanced AI: Agents, RAG, and the Future of Intelligence

Production AI needs explicit routing, memory, execution, and evaluation layers rather than one loop.

AI Architecture Patterns: Routers, Planner-Worker Loops, Memory Layers, and Evaluation Guardrails

datastructures-and-algorithms

Master shortest paths, dependency resolution, and cycle detection with robust Java solutions.

Advanced Graph Algorithms: Dijkstra, Topological Sort, and Disjoint Set Union (DSU)

An LLM can talk, but an AI Agent can *act*. We explain how Agents use the ReAct framework to brow...

AI Agents Explained: When LLMs Start Using Tools

A production-first guide to picking the right vector index for recall, latency, memory, and update patterns.

ANN Index Types Explained: When to Choose Flat, HNSW, IVF, or IVF-PQ

apache-spark-engineering

Process petabytes with Python: understand Spark's execution model, DataFrame transformations, partitioning strategy, and real-time streaming.

Apache Spark for Data Engineers: RDDs, DataFrames, and Structured Streaming

ACID guarantees consistency. BASE guarantees availability. We explain the trade-offs between SQL...

BASE Theorem Explained: How it Stands Against ACID

machine-learning-fundamentals

From the query-key-value formulation to multi-head attention — the mechanism that made transformers dominate AI

Attention Mechanism Explained: How Transformers Learn to Focus

system-design-interview-prep

They all sit in front of your servers. But do you need Nginx, HAProxy, or Kong? We clarify the roles of each component.

API Gateway vs. Load Balancer vs. Reverse Proxy: What's the Difference?

architecture-patterns-for-production-systems

Mobile apps need less data than Desktop dashboards. Why serve them the same JSON? The Backend for Frontend (BFF) pattern solves this.

Backend for Frontend (BFF): Tailoring APIs for UI

BFS explores graphs level by level using a queue — the key to shortest paths and level-order problems.

BFS — Breadth-First Search: Level-by-Level Graph Exploration

Backtracking is a brute-force algorithmic technique that solves problems by trying to build a sol...

Exploring Backtracking Techniques in Data Structures

big-data-engineering

Choose ingestion, serving, and ownership patterns deliberately when data platforms start to scale.

Big Data Architecture Patterns: Lambda, Kappa, CDC, Medallion, and Data Mesh

Why traditional databases fail at scale, what the 5 Vs really mean, and a practical map of the big data ecosystem from ingestion to insights.

Big Data 101: The 5 Vs, Ecosystem, and Why Scale Breaks Everything

Not all Binary Trees are created equal. We explain Full, Complete, Perfect, and Balanced Binary T...

Exploring Different Types of Binary Trees

Master Big O notation: measure algorithm efficiency from O(1) to O(n²) with growth diagrams, worked examples, and the interview mental model.

Big O Notation Explained: Time Complexity, Space Complexity, and Why They Matter in Interviews

Binary search has five distinct patterns — classic, leftmost, rightmost, rotated arrays, and 2D matrices.

Binary Search Patterns: Five Variants Every Senior Engineer Knows

When microservices help, when they hurt, and how to decompose a monolith the right way.

Microservices Architecture: Decomposition, Communication, and Trade-offs

Preserve business consistency across services with explicit write, publish, and compensation flows.

Microservices Data Patterns: Saga, Transactional Outbox, CQRS, and Event Sourcing

Reliable integrations depend on contracts, retries, dedupe, and ownership more than transport alone.

Integration Architecture Patterns: Orchestration, Choreography, Schema Contracts, and Idempotent Receivers

Design independent command and query paths to scale reads without weakening write correctness.

CQRS Pattern: Separating Write Models from Query Models at Scale

Persist domain facts as immutable events and rebuild state predictably under change.

Event Sourcing Pattern: Auditability, Replay, and Evolution of Domain State

Model long-running workflows with compensating actions instead of fragile global transactions.

Saga Pattern: Coordinating Distributed Transactions with Compensation

The Dual Write Problem: Why Two Writes Always Fail Eventually — and How to Fix It

Trip fast on unhealthy dependencies to protect latency and preserve upstream capacity.

Circuit Breaker Pattern: Prevent Cascading Failures in Service Calls

Partition thread, connection, and queue resources so one noisy path cannot starve the system.

Bulkhead Pattern: Isolating Capacity to Protect Critical Workloads

Route failed messages out of hot paths to preserve throughput and enable deterministic replay.

Dead Letter Queue Pattern: Isolating Poison Messages and Recovering Safely

Design and implement cell-based routing using Spring Cloud Gateway to limit system blast radius.

Cell-Based Architectures: Designing Fault Isolation Boundaries for Million-User Apps

Move legacy systems safely by carving seams, translating contracts, and reducing rewrite risk.

Modernization Architecture Patterns: Strangler Fig, Anti-Corruption Layers, and Modular Monoliths

Learn how to evolve relational database schemas under live traffic using the Expand-Contract pattern in Spring Boot.

Zero-Downtime Schema Migrations: The Expand-Contract Pattern for Live Production DBs

Control activation by cohort, tenant, or region without redeploying application code.

Feature Flags Pattern: Decouple Deployments from User Exposure

Release safety depends on traffic control, rollback speed, and separating deploy from exposure.

Deployment Architecture Patterns: Blue-Green, Canary, Shadow Traffic, Feature Flags, and GitOps

## Scenario context

A "one-size-fits-all" API causes bloated mobile payloads and underpowered desktop dashboards. The **Backend for Frontend (BFF)** pattern solves this by creating a dedicated API server for each client type — the mobile BFF reshapes data for small screens, the web BFF aggregates for dashboards.



```architecture
{
  "ok": true,
  "diagram": {
    "version": 1,
    "title": "Backend for Frontend (BFF) Pattern",
    "d…

This walkthrough focuses on **The One-Size-Fits-All API Problem**.

Before making a design choice, clarify:

- You have a mobile app and a web dashboard.
- The problem: - Mobile loads a user profile page.
- The generic /user/{id} endpoint returns 50 fields.

**Concepts in play:** api-gateway, load-balancing, event-driven-architecture, graph-traversal-algorithms

## Stage 1 — BFF Basics: One Backend Per Frontend

- A generic REST API returns the same response shape to every caller.
- That works for simple apps but breaks down when clients have radically different needs: - Mobile apps run on limited bandwidth and battery.
- They need compact, pre-computed responses.
- - Web dashboards need rich, aggregated data combining multiple domain objects.

*Applies **api-gateway** to the decision above.*

**Observable outcome:** - Web dashboards need rich, aggregated data combining multiple domain objects.

## Stage 2 — One API Layer per Client Type

- The BFF pattern introduces a thin, client-specific server between each frontend and the shared microservices: Key properties: - Tight coupling by design: The Mobile BFF is owned by the mobile team.
- When the app screen changes, the BFF changes with it.
- - Data shaping: The BFF projects the microservice response to exactly what the client needs.
- - Protocol bridging: Microservices may speak gRPC internally.

*Applies **load-balancing** to the decision above.*

```mermaid
graph TD
    Mobile[Mobile App] --> MBFF[Mobile BFF]
    Web[Web Dashboard] --> WBFF[Web BFF]

    MBFF --> UserSvc[User Service]
    MBFF --> CartSvc[Cart Service]

    WBFF --> UserSvc
    WBFF --> CartSvc
    WBFF --> RecSvc[Recommendation Service]
    WBFF --> NotifSvc[Notification Service]
```

**Observable outcome:** - Protocol bridging: Microservices may speak gRPC internally.

## Stage 3 — Request Flow Through a BFF

- The request lifecycle from client to microservices flows through the BFF: Each step adds value: 1.
- Auth validation — the BFF verifies the token before hitting any upstream service.
- Parallel upstream calls — fan out to N services simultaneously to minimize latency.
- Response aggregation — merge the N responses into one JSON object.

*Applies **event-driven-architecture** to the decision above.*

```mermaid
flowchart TD
    A[Client Request] --> B{Which client?}
    B -->|Mobile| C[Mobile BFF]
    B -->|Web| D[Web BFF]
    C --> E[Auth Token Validation]
    D --> E
    E --> F[Parallel Upstream Calls]
    F --> G[User Service]
    F --> H[Cart Service]
    F --> I[Rec Service]
    G --> J[Response Aggregation]
    H --> J
    I --> J
    J --> K[Field Projection]
    K --> L[Client Response]
```

```mermaid
sequenceDiagram
  participant MB as Mobile App
  participant MBff as Mobile BFF
  participant WB as Web Dashboard
  participant WBff as Web BFF
  participant US as User Service
  participant CS as Cart Service
  participant RS as Rec Service
  MB->>MBff: GET /mobile/home/{userId}
  par parallel fetch
    MBff->>US: getUser(userId)
    US-->>MBff: firstName only
    MBff->>CS: getCart(userId)
    CS-->>MBff: itemCount only
  end
  MBff-->>MB: compact 8-field response
  WB->>WBff: GET /web/dashboard/{userId}
  par parallel fetch
    WBff->>US: getUser(userId)
    US-->>WBff: full profile
    WBff->>CS: getCart(userId)
    CS-->>WBff: full cart detail
    WBff->>RS: getTopRecs(userId)
    RS-->>WBff: 10 recommendations
  end
  WBff-->>WB: aggregated dashboard payload
```

**Observable outcome:** Response aggregation — merge the N responses into one JSON object.

## Stage 4 — Deep Dive: What Lives Inside a BFF

- A BFF is not a full microservice.
- It is a lightweight orchestration layer.
- Typical responsibilities: A BFF should contain no business logic .
- If you find yourself writing pricing rules or fraud detection inside a BFF, that code belongs in a separate service.

*Applies **graph-traversal-algorithms** to the decision above.*

**Observable outcome:** If you find yourself writing pricing rules or fraud detection inside a BFF, that code belongs in a separate service.

## Stage 5 — Deep Dive: Parallel Aggregation: The Common BFF Pattern

- The most useful thing a BFF does is fan out requests and merge them: Three upstream calls run in parallel (~80 ms total instead of sequential ~240 ms).
- The client gets a single clean response in one round trip.

*Applies **api-gateway** to the decision above.*

```python
# Mobile BFF: home screen endpoint
async def get_home_screen(user_id: str):
    user, cart, recommendations = await asyncio.gather(
        user_service.get(user_id),
        cart_service.get(user_id),
        rec_service.get_top5(user_id),
    )
    return {
        "name": user.first_name,
        "cart_count": len(cart.items),
        "recs": [r.title for r in recommendations],
    }
```

**Observable outcome:** The client gets a single clean response in one round trip.

## Stage 6 — Practical: Adding a New Mobile Screen

- Suppose you add a "notifications center" screen to the mobile app.
- The workflow with a BFF: Step 1 — Identify what the screen needs: - Unread notification count (from Notification Service) - Top 3 notification previews (from Notification Service) - User's preferred notification type (f…
- The web dashboard has its own endpoint for notifications with a different shape.
- This is the BFF promise: frontend-driven API changes stay contained to that frontend's BFF .

*Applies **load-balancing** to the decision above.*

```python
# Mobile BFF: notifications screen endpoint
async def get_notifications_screen(user_id: str):
    notifs, prefs = await asyncio.gather(
        notification_service.get_recent(user_id, limit=3),
        user_service.get_notification_prefs(user_id),
    )
    return {
        "unread_count": notifs.total_unread,
        "previews": [{"id": n.id, "text": n.text[:50]} for n in notifs.items],
        "sound_on": prefs.sound_enabled,
    }
```

**Observable outcome:** This is the BFF promise: frontend-driven API changes stay contained to that frontend's BFF .

## End-to-end outcome

You have a mobile app and a web dashboard.

## Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

## Failure and degraded modes

### Trade-offs & Failure Modes: Trade-offs, Failure Modes & Decision Guide

- **Detection:** BFF is not always the right answer: Code duplication risk: If mobile and web BFFs share 80% of logic, refactor the common parts into shared service clients — not into both BFFs.
- **User or operator impact:** BFF is not always the right answer: Code duplication risk: If mobile and web BFFs share 80% of logic, refactor the common parts into shared service clients — not into both BFFs.



## Scenario context

Microservices let teams deploy and scale services independently — but every service boundary you draw costs you a network hop, a consistency challenge, and an operational burden. The architecture pays off only when your team and traffic scale have genuinely outgrown a well-structured monolith.

> **Interview signal:** Know the decomposition strategies (DDD bounded contexts, Strangler Fig), distinguish sync from asyn…

This walkthrough focuses on **Deep Dive: Database-per-Service and Why Shared Data Breaks Independent Deployments**.

Before making a design choice, clarify:

- The database-per-service pattern is often the hardest constraint for teams migrating from a monolith, because shared databases feel efficient.
- They are efficient — until they become a coupling point that makes every schema migration a cross-team coordination exercise.
- Internals: How the Database-per-Service Contract Works The core rule is strict: no service may read from or write to another service's database .

**Concepts in play:** api-gateway, caching, design-patterns-gof, consistency-replication

## Stage 1 — The Monolith That Took Down Netflix for Three Days

- In 2008, a database corruption event at Netflix took their entire DVD-rental and streaming platform offline for three days.
- The root cause was not the database itself — it was the monolithic architecture.
- Every feature, every team's code, every data access pattern was deployed as a single unit and connected to a single database.
- When that database went down, everything went down at once.

*Applies **api-gateway** to the decision above.*

**Observable outcome:** When that database went down, everything went down at once.

## Stage 2 — The Four Decomposition Strategies That Actually Work

- Getting service boundaries right is the most consequential decision in a microservices migration.
- Cut them wrong and you get a distributed monolith; cut them right and you get genuine deployment independence.
- Domain-Driven Design and Bounded Contexts Domain-Driven Design (DDD) is the most principled approach.
- A bounded context is a clear boundary within which a domain model applies consistently.

*Applies **caching** to the decision above.*

**Observable outcome:** A bounded context is a clear boundary within which a domain model applies consistently.

## Stage 3 — From Monolith to Microservices: The Decomposition Flow

- The diagram below shows a monolith being incrementally extracted using the Strangler Fig pattern, with each bounded context becoming an independent service behind a shared API gateway.
- Reading the diagram: The API gateway acts as the Strangler Fig facade.
- Extracted services (User, Order, Notification) each own a dedicated database and receive traffic directly.
- The monolith still handles the remaining unextracted features and its shared legacy database — the Step 3: in progress state.

*Applies **design-patterns-gof** to the decision above.*

```mermaid
graph TD
    Client([Client]) --> GW[API Gateway / Facade]

    GW --> |"Step 1: extracted"| US[User Service]
    GW --> |"Step 2: extracted"| OS[Order Service]
    GW --> |"Step 3: in progress"| MONO[Monolith legacy features]
    GW --> |"Step 4: extracted"| NS[Notification Service]

    US --> UDB[(User DB Postgres)]
    OS --> ODB[(Order DB MySQL)]
    NS --> NDB[(Notification DB Redis)]
    MONO --> MDB[(Shared Legacy DB)]

    style MONO fill:#f9c74f,stroke:#f3722c,color:#000
    style GW fill:#4cc9f0,stroke:#4361ee,color:#000
    style MDB fill:#f9c74f,stroke:#f3722c,color:#000

```

**Observable outcome:** The monolith still handles the remaining unextracted features and its shared legacy database — the Step 3: in progress state.

## Stage 4 — How Services Talk to Each Other: Sync, Async, and Discovery

- Once services are split, they need to communicate.
- The choice between synchronous and asynchronous communication is one of the most impactful decisions in microservices design.
- The diagram below shows a typical microservices deployment — an API gateway at the entry point, independent business capability services in the middle tier, and each service owning its isolated data store.
- Async side effects (like notifications) route through a message broker rather than blocking the caller.

*Applies **consistency-replication** to the decision above.*

```mermaid
flowchart TD
    Client["Client (Browser or Mobile)"] --> GW["API Gateway (Auth, Routing, Rate Limiting)"]
    GW --> SvcA["Order Service"]
    GW --> SvcB["User Service"]
    GW --> SvcC["Inventory Service"]
    SvcA --> DBA[("Order DB (MySQL)")]
    SvcB --> DBB[("User DB (PostgreSQL)")]
    SvcC --> DBC[("Inventory DB (Redis)")]
    SvcA -->|"async event"| MQ["Message Broker (Kafka)"]
    MQ --> SvcD["Notification Service"]
    SvcD --> DBD[("Notification DB (Redis)")]
```

**Observable outcome:** Async side effects (like notifications) route through a message broker rather than blocking the caller.

## Stage 5 — Real-World Applications: Netflix's 700 Services and Segment's Reversal

- Two production case studies at opposite ends of the microservices spectrum show that the architecture is a tool, not a destination.
- Netflix (2008–2016): 1 service → 700+ services.
- After the 2008 database outage, Netflix had a concrete and expensive problem: one failure domain took down everything.
- Their traffic was also growing 10x year-over-year.

*Applies **api-gateway** to the decision above.*

**Observable outcome:** Their traffic was also growing 10x year-over-year.

## Stage 6 — Spring Cloud: How It Wires Microservices Together

- Spring Cloud is the de-facto Java framework for microservices infrastructure.
- It provides service discovery (Eureka), configuration management (Config Server), API gateway (Spring Cloud Gateway), and resilience (Resilience4j circuit breakers) — all integrating with Spring Boot applications via an…
- Service Registration with Eureka Each service registers with Eureka on startup by declaring its application name and the Eureka server URL.
- The client pulls the live registry at startup and polls for updates at a configurable interval, so it always has a fresh list of available instances for each dependency.

*Applies **caching** to the decision above.*

**Observable outcome:** The client pulls the live registry at startup and polls for updates at a configurable interval, so it always has a fresh list of available instances for each dependency.

## End-to-end outcome

After working through the stages, you should be able to explain:

- Service boundaries set in year one are hard to change.
- Wrong decomposition creates tight coupling at the network layer instead of the code layer.

## Alternatives considered

### Synchronous vs Asynchronous Communication Flow

Reading the diagram: The checkout path is synchronous — the Order Service blocks on inventory validation because correctness requires it.

### Decision Guide: When Microservices Help and When They Hurt

The Segment reversal happened because they decomposed before the organizational pressure existed.

### Decision Point: Synchronous vs Asynchronous in Practice

Decision Point: Synchronous vs Asynchronous in Practice


## Failure and degraded modes

### Trade-offs and Failure Modes: Cascade Failures and the Patterns That Contain Them

- **Detection:** The most dangerous failure mode in microservices is the cascade.
- **User or operator impact:** The most dangerous failure mode in microservices is the cascade.

### Microservices vs Monolith: The Trade-offs Nobody Puts on the Architecture Slide

- **Detection:** The column you should read first in an interview: Deployment complexity and Operational overhead .
- **User or operator impact:** The column you should read first in an interview: Deployment complexity and Operational overhead .



## Scenario context

Microservices get risky when teams distribute writes without defining how business invariants survive network delays, retries, and partial failures. Patterns like transactional outbox, saga, CQRS, and event sourcing exist to make those rules explicit.

> **TLDR**: The core challenge is not splitting services. It is deciding where truth lives, how changes propagate, and how failure is compensated.

A logistics startu…

This walkthrough focuses on **Why Data Patterns Become the Hard Part of Microservices**.

Before making a design choice, clarify:

- Teams often discuss microservices in terms of deploy independence or team ownership.
- Those are valid benefits, but the hardest engineering work starts after the split.
- A single database transaction no longer covers the full business workflow.

**Concepts in play:** event-driven-architecture, transaction-isolation-distributed-acid, jvm-internals-and-garbage-collection, consistency-replication

## Stage 1 — Comparing Transactional Outbox, Saga, CQRS, and Event Sourcing

- The patterns solve adjacent but different problems.
- The crucial insight is that these patterns layer.
- An order service may use database-per-service plus transactional outbox.
- A saga may orchestrate several services.

*Applies **event-driven-architecture** to the decision above.*

```mermaid
flowchart TD
 Q{Distributed write needed?}
 Q -->|Single service owns| A[Local transaction]
 Q -->|Multi-service| B{Strong consistency needed?}
 B -->|Yes, short-lived| C[2PC / XA]
 B -->|No, compensatable| D[Saga pattern]
 C --> E[Blocking locks, low throughput]
 D --> F[Async, compensating actions]
 D --> G[Choreography or Orchestration]
```

**Observable outcome:** A saga may orchestrate several services.

## Stage 2 — How the Write Path Works in a Distributed Workflow

- A safe microservices write path typically follows this structure: 1.
- A command enters the owning service.
- The service validates business rules and writes local state.
- The same local transaction writes an outbox record.

*Applies **transaction-isolation-distributed-acid** to the decision above.*

```mermaid
sequenceDiagram
 participant Svc as OrderService
 participant DB as ServiceDB
 participant OB as OutboxTable
 participant CDC as Debezium CDC
 participant K as Kafka
 participant Cons as InventoryService
 Svc->>DB: Write order row
 Svc->>OB: Write outbox row
 Note over DB,OB: Same transaction
 CDC->>OB: Tail transaction log
 CDC->>K: Publish OrderPlaced
 K->>Cons: Deliver event
 Cons->>Cons: Reserve stock
```

**Observable outcome:** The same local transaction writes an outbox record.

## Stage 3 — Workflow Pattern Flow: Order, Outbox, Saga, and Read Model

- This sequence shows the important separation: local consistency is strict, cross-service consistency is coordinated through events and compensation.

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

```mermaid
flowchart TD
 A[Client sends create order command] --> B[Order Service validates business rules]
 B --> C[Order DB commits order and outbox record]
 C --> D[Outbox relay publishes OrderCreated]
 D --> E[Payment Service processes payment]
 D --> F[Inventory Service reserves stock]
 E --> G[Payment result event]
 F --> H[Inventory result event]
 G --> I[Order status read model updates]
 H --> I
```

**Observable outcome:** This sequence shows the important separation: local consistency is strict, cross-service consistency is coordinated through events and compensation.

## Stage 4 — Real-World Applications: Checkout, Booking, and Subscription Billing

- Checkout systems are the classic saga example because no single service owns the full workflow.
- Orders, payments, inventory, notifications, and shipment planning all have different correctness constraints.
- Booking systems benefit from outbox plus CQRS because users often need fast status views while the write path remains tightly controlled.
- Read models can summarize booking state for portals and support tools without weakening the write model.

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** Read models can summarize booking state for portals and support tools without weakening the write model.

## Stage 5 — Practical Example: Designing an Order Workflow

- Suppose an order service owns order creation but payment and inventory live elsewhere.
- persist the order and an outbox event in one local transaction, 2.
- let payment and inventory act independently, 4.
- track saga state in an orchestration layer or durable workflow log, 5.

*Applies **event-driven-architecture** to the decision above.*

**Observable outcome:** track saga state in an orchestration layer or durable workflow log, 5.

## Stage 6 — Operator Field Note: What Fails First in Production

- A recurring pattern from postmortems is that incidents in Microservices Data Patterns: Saga, Transactional Outbox, CQRS, and Event Sourcing start with weak signals long before full outage.
- - Early warning signal: one guardrail metric drifts (error rate, lag, divergence, or stale-read ratio) while dashboards still look mostly green.
- - First containment move: freeze rollout, route to the last known safe path, and cap retries to avoid amplification.
- - Escalate immediately when: customer-visible impact persists for two monitoring windows or recovery automation fails once.

*Applies **transaction-isolation-distributed-acid** to the decision above.*

```yaml
runbook:
 pattern: '2026-03-13-microservices-data-patterns-saga-outbox-cqrs-and-event-sourcing'
 checks:
 - nam
e: primary_guardrail
 query: 'error_rate OR drift_rate OR divergence_rate'
 threshold: 'breach_for_2_windows'
 - nam
e: rollback_readiness
 query: 'last_successful_drill_age_minutes'
 threshold: '<= 10080'
 action_on_breach:
 - freeze_rollout
 - route_to_safe_path
 - page_owner
```

**Observable outcome:** - Escalate immediately when: customer-visible impact persists for two monitoring windows or recovery automation fails once.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - Distributed writes need an explicit consistency story.
- - Transactional outbox is the safest default for reliable event emission.

## Alternatives considered

### Decision Guide: Which Pattern Should You Apply?

Start from business invariants, not pattern popularity.


## Failure and degraded modes

### The Internals: Ordering, Idempotency, Read Models, and Compensation

- **Detection:** Transactional outbox exists because writing local state and publishing an event cannot be treated as two unrelated operations.
- **User or operator impact:** Transactional outbox exists because writing local state and publishing an event cannot be treated as two unrelated operations.

### Performance Analysis: Replay Cost, Lag Budgets, and Hot Aggregates

- **Detection:** Snapshotting is often necessary in event-sourced systems because replaying thousands of events for a hot aggregate on every request is wasteful.
- **User or operator impact:** Snapshotting is often necessary in event-sourced systems because replaying thousands of events for a hot aggregate on every request is wasteful.

### Trade-offs & Failure Modes: Trade-offs and Failure Modes

- **Detection:** The main trade-off is simplicity versus explicitness.
- **User or operator impact:** The main trade-off is simplicity versus explicitness.

## Scenario context

Integration failures usually come from weak contracts, unsafe retries, and missing ownership rather than from choosing the wrong transport. Orchestration, choreography, schema contracts, and idempotent receivers are patterns for making cross-boundary behavior explicit.

> **TLDR**: Integration architecture succeeds when boundaries are explicit: contract first, idempotency by default, bounded retries, and clear workf…

This walkthrough focuses on **Real-World Applications: Realistic Scenario: Payment Callback and ERP Sync**.

Before making a design choice, clarify:

- Stripe: Idempotency Keys and Bounded Retries Stripe delivers 200M+ webhook events/day with at-least-once guarantees.
- After the 2016 amplification incident, they designed retry logic with true exponential backoff (1s → 2s → 4s → ...
- up to 3 days), a hard cap of 25 retries , and a mandatory Idempotency-Key header.

**Concepts in play:** event-driven-architecture, graph-traversal-algorithms, circuit-breaker, database-indexing

## Stage 1 — When to Use Orchestration, Choreography, Contracts, and Idempotent Receivers

- Quick decision rule - External partner callbacks: idempotency + contract checks first.
- - Ordered business process: orchestration first.
- - Broad internal event mesh: choreography + strict contract governance.
- 📊 Orchestration: Central Coordinator This sequence diagram illustrates the orchestration pattern: a central Orchestrator drives the workflow by calling ServiceA, ServiceB, and ServiceC in sequence, passing results down…

*Applies **event-driven-architecture** to the decision above.*

```mermaid
sequenceDiagram
 participant O as Orchestrator
 participant A as ServiceA
 participant B as ServiceB
 participant C as ServiceC
 O->>A: Call ServiceA
 A-->>O: Result A
 O->>B: Call ServiceB (with A result)
 B-->>O: Result B
 O->>C: Call ServiceC (with B result)
 C-->>O: Result C
 Note over O: Aggregates all results
```

**Observable outcome:** 📊 Orchestration: Central Coordinator This sequence diagram illustrates the orchestration pattern: a central Orchestrator drives the workflow by calling ServiceA, ServiceB, and ServiceC in sequence, passing results down…

## Stage 2 — How Reliable Integration Works End-to-End

- Ingress validates signature/auth and schema version.
- System checks idempotency key before side effects.
- Workflow is orchestrated or evented according to ownership model.
- Retries run with backoff and cap.

*Applies **graph-traversal-algorithms** to the decision above.*

**Observable outcome:** Retries run with backoff and cap.

## Stage 3 — How to Implement: Integration Hardening Checklist

- Define canonical envelope fields and contract versioning policy.
- Add schema validation at ingress (before business logic).
- Implement idempotency store with retention policy.
- Configure exponential backoff with max attempts.

*Applies **circuit-breaker** to the decision above.*

**Observable outcome:** Configure exponential backoff with max attempts.

## Stage 4 — Deep Dive: Contract Drift and Duplicate Safety Internals

- The Internals: Dedupe Keys, Ownership, and Replay Semantics Idempotency is stateful.
- The receiver must store processed IDs durably and define key scope clearly (global message ID vs partner+operation+business key).
- Replay semantics should be explicit: - replay for recovery, - replay for backfill, - replay for audit.
- Each mode may have different side-effect rules.

*Applies **database-indexing** to the decision above.*

**Observable outcome:** Each mode may have different side-effect rules.

## Stage 5 — Integration Flow: Safe External Callback Boundary

- This flowchart traces the safe ingestion path for an external webhook or API call: authentication and signature verification at the gateway, schema and version validation, then an idempotency lookup that short-circuits…
- New events are persisted as canonical records before being forwarded to the orchestrator or event bus, and failures trigger retry-with-backoff then dead-letter routing.
- The takeaway is that every external boundary crossing requires layered validation — no partner payload should reach business logic without passing all three checks.

*Applies **event-driven-architecture** to the decision above.*

```mermaid
flowchart TD
 A[Partner webhook or API call] --> B[Gateway auth and signature check]
 B --> C[Schema and version validator]
 C --> D[Idempotency lookup]
 D --> E{Already processed?}
 E -->|Yes| F[Return success idempotently]
 E -->|No| G[Persist canonical event]
 G --> H[Orchestrator or event bus]
 H --> I[Consumer side effects]
 I --> J{Failure?}
 J -->|Yes| K[Retry with backoff then DLQ]
 J -->|No| L[Emit completion with correlation ID]
```

**Observable outcome:** The takeaway is that every external boundary crossing requires layered validation — no partner payload should reach business logic without passing all three checks.

## Stage 6 — Practical Example: Minimal Idempotent Receiver Contract

- Python: Idempotent Webhook Receiver with Retry and Dead-Letter Handling Required envelope fields (Slack/Stripe production pattern): - event id — stable producer-assigned key; never reuse across distinct events - event t…
- Validate contract before side effects.
- Persist dedupe key transactionally with the business outcome.
- Return idempotent success for duplicates — same response, no work done.

*Applies **graph-traversal-algorithms** to the decision above.*

```python
import asyncio, logging
from typing import Optional

logger = logging.getLogger(__name__)

# ── Idempotency store (Postgres-backed, 72-hour retention) ───────────────────
async def is_already_processed(conn, source: str, event_id: str) -> bool:
 row = await conn.fetchrow(
 "SELECT 1 FROM processed_events "
 "WHERE source=$1 AND event_id=$2 AND processed_at > NOW() - INTERVAL '72 hours'",
 source, event_id
 )
 return row is not None

async def mark_processed(conn, source: str, event_id: str, result: dict):
 await conn.execute(
 "INSERT INTO processed_events (source, event_id, result, processed_at) "
 "VALUES ($1, $2, $3, NOW()) ON CONFLICT DO NOTHING",
 source, event_id, str(result)
 )

# ── Idempotent handler with exponential backoff + DLQ routing ────────────────
async def handle_webhook(event: dict, conn, dlq_publish) -> dict:
 source = event.get("source", "unknown")
 event_id = event.get("event_id")
 corr_id = event.get("correlation_id")

 if not event_id:
 raise ValueError("event_id required — cannot guarantee idempotency")

 if await is_already_processed(conn, source, event_id):
 logger.info(f"Duplicate skipped: {source}/{event_id} [{corr_id}]")
 return {"status": "duplicate_skipped", "event_id": event_id}

 max_attempts = 3
 for attempt in range(max_attempts):
 try:
 result = await apply_business_logic(event)
 await mark_processed(conn, source, event_id, result)
 return {"status": "processed", "event_id": event_id}
 except Exception as exc:
 backoff = 2 ** attempt # 1s → 2s → 4s (Stripe retry pattern)
 if attempt < max_attempts - 1:
 logger.warning(f"Retry {attempt+1}/{max_attempts} in {backoff}s: {exc}")
 await asyncio.sleep(backoff)
 else:
 logger.error(f"Exhausted retries, routing to DLQ: {event_id}")
 await dlq_publish(event, error=str(exc), correlation_id=corr_id)
 return {"status": "dlq_routed", "event_id": event_id}

async def apply_business_logic(event: dict) -> dict:
 return {"processed": True, "type": event.get("event_type")}
```

**Observable outcome:** Return idempotent success for duplicates — same response, no work done.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - Protocol choice matters less than contract and replay discipline.
- - Orchestration and choreography are complementary, not mutually exclusive.

## Alternatives considered

### Decision Guide: Choosing Integration Style Quickly

If replay behavior is undefined, pause architecture changes until that is clarified.


## Failure and degraded modes

### Why Integration Failures Are Usually Boundary Failures

- **Detection:** Most integration incidents are not protocol failures.
- **User or operator impact:** Most integration incidents are not protocol failures.

### Trade-offs & Failure Modes: Pros, Cons, and Risks

- **Detection:** 📊 Choreography: Event-Driven Flow This sequence diagram contrasts with orchestration by showing a choreography model: ServiceA publishes EventA to an EventBus, ServiceB reacts autonomously and publi…
- **User or operator impact:** 📊 Choreography: Event-Driven Flow This sequence diagram contrasts with orchestration by showing a choreography model: ServiceA publishes EventA to an EventBus, ServiceB reacts autonomously and publi…

Start with separation, API tailoring, and bounded architecture decisions.

Core Architecture Patterns

## Scenario context

CQRS works when read and write workloads diverge, but only with explicit freshness budgets and projection reliability. The hard part is not separating models — it is operating lag, replay, and rollback safely.



```architecture
{
 "ok": true,
 "diagram": {
 "version": 1,
 "title": "CQRS Pattern: Separating Read and Write Models",
 "description": "This architecture illustrates the CQRS pattern, separating…

This walkthrough focuses on **Why CQRS Exists: Protect Write Truth While Shaping Fast Reads**.

Before making a design choice, clarify:

- Teams usually reach for CQRS after the same failure pattern repeats: one transactional model is asked to serve invariants, dashboards, search, timelines, and external API reads all at once.
- The result is often lock contention on the write path, expensive joins on the read path, and emergency cache layers that hide freshness problems instead of fixing them.
- In architecture reviews, CQRS should answer four operational questions before anyone sketches a read model: - Which business rules must stay synchronous on the write path?

**Concepts in play:** event-driven-architecture, api-gateway, caching, transaction-isolation-distributed-acid

## Stage 1 — The Boundary Model: Command Side, Event Stream, Projection Side

- At a practical level, CQRS is not just two databases.
- It is a contract about where truth is written and how derivative views are built.
- 📊 Command-Query Split: Write Side vs Read Side This flowchart draws the CQRS architectural boundary in its simplest form: commands flow through a handler to a write store, and a projection worker asynchronously propaga…
- The critical visual detail is that the outbox or CDC mechanism sits at the boundary between the write and read subgraphs, making the write-to-read propagation explicit and decoupled.

*Applies **event-driven-architecture** to the decision above.*

```mermaid
flowchart LR
 subgraph Write Side
 CMD[Command] --> CH[Command Handler]
 CH --> WDB[(Write Store)]
 WDB --> OBX[Outbox / CDC]
 end
 subgraph Read Side
 OBX --> PW[Projection Worker]
 PW --> RDB[(Query Store)]
 RDB --> QH[Query Handler]
 QH --> QRY[Query Response]
 end
```

**Observable outcome:** The critical visual detail is that the outbox or CDC mechanism sits at the boundary between the write and read subgraphs, making the write-to-read propagation explicit and decoupled.

## Stage 2 — How the Write Path and Read Path Stay Separate

- A healthy CQRS flow looks boring on purpose: 1.
- The command handler validates the requested state change against current write-side rules.
- The transaction commits the write and records an outbox event or change record in the same durability boundary.
- A relay publishes that committed event to projection workers.

*Applies **api-gateway** to the decision above.*

```mermaid
sequenceDiagram
 participant C as Client
 participant CA as Command API
 participant CH as Command Handler
 participant WS as Write Store
 participant OB as Outbox
 participant PW as Projection Worker
 C->>CA: POST /orders (command)
 CA->>CH: validate and handle
 CH->>WS: commit to write store
 WS->>OB: record outbox event (atomic)
 OB->>PW: relay committed event
 PW->>PW: update read model with checkpoint
 CA-->>C: 202 Accepted
```

```mermaid
sequenceDiagram
 participant C as Client
 participant QA as Query API
 participant RM as Read Model
 participant QS as Query Store
 C->>QA: GET /orders/timeline
 QA->>RM: query read model
 RM->>QS: fetch from query store
 QS-->>RM: projection data
 RM-->>QA: shaped response
 QA-->>C: timeline response (may lag write by seconds)
```

**Observable outcome:** A relay publishes that committed event to projection workers.

## Stage 3 — Deep Dive: Lag, Replay, and Projection Safety

- The Internals: Write Authority, Checkpoints, and Read Staleness The write side is the only place where business invariants should be enforced.
- Projection workers are downstream materializers; they should never be asked to resolve conflicts that belong to the command model.
- A replayed projection should rebuild a view from committed events, not guess what the latest truth is.
- Operators usually need three durable markers: - a source-of-truth commit version or event ID, - a per-projection checkpoint, - a freshness budget that maps lag into user impact.

*Applies **caching** to the decision above.*

**Observable outcome:** Operators usually need three durable markers: - a source-of-truth commit version or event ID, - a per-projection checkpoint, - a freshness budget that maps lag into user impact.

## Stage 4 — Operator Field Note: Freshness Budgets Fail Before Correctness Does

- In incident reviews, the first visible symptom of CQRS trouble is usually outdated reads, not corrupted writes.
- Support tickets say a status is old long before anyone proves the write path is wrong.
- Operators usually find that the most valuable architecture review artifact is a freshness table per read surface: acceptable lag, pager threshold, and replay procedure.

*Applies **transaction-isolation-distributed-acid** to the decision above.*

**Observable outcome:** Operators usually find that the most valuable architecture review artifact is a freshness table per read surface: acceptable lag, pager threshold, and replay procedure.

## Stage 5 — CQRS Flow: Commit Once, Project Many

- This flowchart shows the full CQRS data pipeline from a single command API entry point through transactional write, outbox relay, and projection worker to multiple specialized query stores.
- Two additional outputs — a committed version token returned to the read path and a projection checkpoint — make the consistency and replay properties of the pipeline explicit.
- The key takeaway is that a single committed write fans out to as many read models as the system needs, each optimized independently, all derived from the same immutable source of truth in the write store.

*Applies **event-driven-architecture** to the decision above.*

```mermaid
flowchart TD
 A[Command API] --> B[Validate business rule]
 B --> C[Transactional write store]
 C --> D[Outbox or change stream]
 D --> E[Projection worker]
 E --> F[Query store]
 F --> G[API or UI read path]
 E --> H[Projection checkpoint]
 C --> I[Committed version token]
 I --> G
```

**Observable outcome:** The key takeaway is that a single committed write fans out to as many read models as the system needs, each optimized independently, all derived from the same immutable source of truth in the write store.

## Stage 6 — Practical Example: Order Service With Axon Framework

- The order scenario from the previous section maps directly to Axon's programming model.
- Commands flow in through CommandGateway , the aggregate persists events to the event store, and the projection worker builds the timeline read model asynchronously.
- Maven dependency Commands and queries as plain records Commands carry intent; queries carry selection criteria.
- Write side: aggregate enforces invariants, never touches the read store AggregateLifecycle.apply() commits the event to the Axon event store.

*Applies **api-gateway** to the decision above.*

```mermaid
flowchart LR
 A[REST POST /orders] --> B[CommandGateway]
 B --> C[OrderAggregate]
 C --> D[EventStore]
 D --> E[EventBus]
 E --> F[@EventHandler Projection Worker]
 F --> G[QueryStore]
 G --> H[QueryGateway]
 H --> I[REST GET /timeline]
```

```xml
<dependency>
 <groupId>org.axonframework</groupId>
 <artifactId>axon-spring-boot-starter</artifactId>
 <version>4.9.3</version>
</dependency>
```

**Observable outcome:** Write side: aggregate enforces invariants, never touches the read store AggregateLifecycle.apply() commits the event to the Axon event store.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - CQRS is a control boundary between write truth and read convenience, not a blanket microservices rule.
- - Freshness budgets and replay tooling matter as much as the schema design.

## Alternatives considered

### Decision Guide: When CQRS Earns Its Complexity

Start with one domain that already hurts, such as orders or billing, and prove that lag, replay, and recovery are manageable before expanding the pattern elsewhere.


## Failure and degraded modes

### Trade-offs & Failure Modes: Trade-offs and Failure Modes

- **Detection:** CQRS is worth the cost when read workloads are truly different.
- **User or operator impact:** CQRS is worth the cost when read workloads are truly different.

## Scenario context

Event sourcing pays off when regulatory audit history and replay are first-class requirements — but it demands strict schema evolution, a snapshot strategy, and a framework that owns aggregate lifecycle. Spring Boot + Axon Framework is the fastest production-grade path on the JVM.

In 2017, a GitLab database administrator ran `rm -rf` on the wrong production server. They had no event log — just nightly snapshots. Si…

This walkthrough focuses on **Why Storing Events Instead of State Changes Everything**.

Before making a design choice, clarify:

- In 2017, a GitLab database administrator ran rm -rf on the wrong production server.
- They had no event log — just nightly snapshots.
- Six hours of user data was lost permanently, and thousands of repositories were irrecoverable.

**Concepts in play:** event-driven-architecture, jvm-internals-and-garbage-collection, consistency-replication, design-principles-solid

## Stage 1 — The Four Building Blocks of an Event-Sourced System

- Every production event-sourced system has four roles: 1.
- Command — an intent to change state; validated against current aggregate state before writing.
- Aggregate — the consistency boundary; enforces invariants, emits events, and advances its internal state machine.
- Event Store — the append-only log; events are immutable, each aggregate instance owns a stream by ID.

*Applies **event-driven-architecture** to the decision above.*

**Observable outcome:** Event Store — the append-only log; events are immutable, each aggregate instance owns a stream by ID.

## Stage 2 — How a Command Flows into an Auditable Event Stream

- Solid arrows show the live command path.
- Dashed arrows show replay — the TrackingEventProcessor resets its token to reconstruct the query model for audit at any historical timestamp.
- The aggregate never writes directly to the query model.
- It emits events; projections consume them independently.

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

```mermaid
flowchart TD
 C[Client Command] --> CH["Command Handler (SubscriptionAggregate)"]
 CH -->|"validates invariants applies event"| ES[(Event Store Append-Only Log)]
 ES -->|"event published on event bus"| P["BillingHistoryProjection (Event Handler)"]
 P --> QM[(Query Model BillingHistoryRepository)]
 QM -->|"query response"| Q[GetBillingHistoryQuery]

 ES -. "token-based replay" .-> RP["Replay Processor (TrackingEventProcessor)"]
 RP -. "rebuilds view for audit dispute" .-> QM

 style ES fill:#f5f5f5,stroke:#555
 style QM fill:#e8f4e8,stroke:#555
 style RP fill:#fff3e0,stroke:#f90,stroke-dasharray: 5 5
```

**Observable outcome:** It emits events; projections consume them independently.

## Stage 3 — Deep Dive: Inside the Aggregate: State Machines, Snapshots, and Schema Evolution

- Internals: Aggregate State Reconstruction An aggregate's state exists only in memory during command processing.
- Before handling a command, the framework loads the aggregate by replaying every past event for that aggregate ID in sequence.
- Each @EventSourcingHandler method advances internal state — status flags, counters, IDs — until the aggregate is fully current.
- The command handler then checks invariants against that reconstructed in-memory state.

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** The command handler then checks invariants against that reconstructed in-memory state.

## Stage 4 — Axon Framework and EventStoreDB: Event Sourcing on the JVM

- Axon Framework is a Spring Boot-native Java framework that manages the full event-sourcing lifecycle: aggregate command handling, event persistence, snapshotting, replay, upcasting, and projection tracking.
- EventStoreDB is a purpose-built append-only database with server-side projections and persistent subscription support — the recommended backend for production Axon deployments requiring audit-grade storage.
- These tools solve the event-sourcing problem by owning the infrastructure that makes aggregates deterministic: Axon's @CommandHandler / @EventSourcingHandler pattern enforces the strict separation between command valida…
- Teams write domain logic; Axon owns the replay machinery.

*Applies **design-principles-solid** to the decision above.*

```xml
<dependency>
 <groupId>org.axonframework</groupId>
 <artifactId>axon-spring-boot-starter</artifactId>
 <version>4.9.3</version>
</dependency>

<dependency>
 <groupId>org.axonframework.extensions.eventstored</groupId>
 <artifactId>axon-eventstoredb-spring-boot-starter</artifactId>
 <version>0.1.0</version>
</dependency>
```

**Observable outcome:** Teams write domain logic; Axon owns the replay machinery.

## Stage 5 — Real-World Applications

- Event sourcing earns its complexity where audit trails and replay are first-class business requirements.

*Applies **event-driven-architecture** to the decision above.*

**Observable outcome:** Event sourcing earns its complexity where audit trails and replay are first-class business requirements.

## Stage 6 — Subscription Billing: Building the Aggregate, Projection, and Replay

- Scenario: A billing platform tracks the lifecycle of each subscription — CREATED → ACTIVATED → SUSPENDED → CANCELLED .
- Every state transition is an immutable domain event appended to the subscription's event stream.
- When a customer disputes a charge, the support team replays the event stream to reconstruct exactly what the account looked like at the moment of the disputed transaction.
- Maven Dependency Domain Events (Immutable Value Objects) Each event is a value object with no setters.

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

```xml
<dependency>
 <groupId>org.axonframework</groupId>
 <artifactId>axon-spring-boot-starter</artifactId>
 <version>4.9.3</version>
</dependency>
```

```java
public record SubscriptionCreatedEvent(
 String subscriptionId, String tenantId, String planId, Instant occurredAt) {}

public record SubscriptionActivatedEvent(
 String subscriptionId, String tenantId, Instant occurredAt) {}

public record SubscriptionSuspendedEvent(
 String subscriptionId, String tenantId, String reason, Instant occurredAt) {}

public record SubscriptionCancelledEvent(
 String subscriptionId, String tenantId, String reason, Instant occurredAt) {}
```

**Observable outcome:** Maven Dependency Domain Events (Immutable Value Objects) Each event is a value object with no setters.

## End-to-end outcome

In 2017, a GitLab database administrator ran rm -rf on the wrong production server.

## Alternatives considered

### Decision Guide: When Event Sourcing Earns Its Complexity

Decision Guide: When Event Sourcing Earns Its Complexity


## Failure and degraded modes

### Trade-offs & Failure Modes in Practice

- **Detection:** Trade-offs & Failure Modes in Practice
- **User or operator impact:** Trade-offs & Failure Modes in Practice

## Scenario context

A Saga replaces fragile distributed 2PC with a sequence of local transactions, each backed by an explicit compensating transaction. Use orchestration when workflow control needs a single brain; use choreography when services must stay loosely coupled.






```mermaid
flowchart TD
 S([Start]) --> INV[ReserveInventory]
 INV -->|InventoryReservedEvent| PAY[AuthorizePayment]
 INV -->|InventoryReservationFailed…

This walkthrough focuses on **Deep Dive: Saga Internals and Failure Semantics**.

Before making a design choice, clarify:

- Compensating Transactions Are Not Rollbacks A compensating transaction is a new, forward-moving transaction that reverses the business effect of a prior step — it is not a database rollback.
- This distinction has real implications: - ReleaseInventoryCommand tells the inventory service "release reservation R-4892 ." The reservation record is not deleted; it is marked released with a timestamp and reason.
- - Compensations must be idempotent : if ReleaseInventoryCommand is delivered twice (broker retry, network hiccup), the second delivery must be a safe no-op, not a double-release.

**Concepts in play:** event-driven-architecture, graph-traversal-algorithms, jvm-internals-and-garbage-collection, message-queues

## Stage 1 — Java Frameworks for Saga Orchestration

- Eventuate Tram: Saga Coordination Without a Dedicated Framework Eventuate Tram (a lightweight saga/transactional messaging library by Chris Richardson) implements sagas as SagaDefinition objects with explicit step seque…
- SagaDefinition chains steps with .withCompensation() — Eventuate Tram executes compensations in reverse step order automatically on failure.
- No central orchestrator wakeup loop; the framework manages message dispatch and state persistence via JDBC.
- For a full deep-dive on Eventuate Tram saga state management and transactional messaging patterns, a dedicated follow-up post is planned.

*Applies **event-driven-architecture** to the decision above.*

```java
public class CheckoutSagaDefinition implements SimpleSaga<CheckoutSagaData> {

 private SagaDefinition<CheckoutSagaData> sagaDefinition =
 step()
 .invokeParticipant(this::reserveInventory)
 .withCompensation(this::releaseInventory)
 .step()
 .invokeParticipant(this::authorizePayment)
 .withCompensation(this::refundPayment)
 .step()
 .invokeParticipant(this::createShipment)
 .build();

 @Override
 public SagaDefinition<CheckoutSagaData> getSagaDefinition() { return sagaDefinition; }

 private CommandWithDestination reserveInventory(CheckoutSagaData data) {
 return send(new ReserveInventoryCommand(data.getOrderId(), data.getItems()))
 .to("inventoryService").build();
 }
 private CommandWithDestination releaseInventory(CheckoutSagaData data) {
 return send(new ReleaseInventoryCommand(data.getOrderId(), data.getReservationId()))
 .to("inventoryService").build();
 }
 // authorizePayment, refundPayment, createShipment follow the same pattern
}
```

```xml
<dependency>
 <groupId>org.axonframework</groupId>
 <artifactId>axon-spring-boot-starter</artifactId>
 <version>4.9.3</version>
</dependency>
```

**Observable outcome:** For a full deep-dive on Eventuate Tram saga state management and transactional messaging patterns, a dedicated follow-up post is planned.

## Stage 2 — Orchestration in Code: The Axon Checkout Saga

- The saga class below is the single source of truth for the checkout workflow.
- Axon persists its fields between events, so the saga survives service restarts mid-flight without losing state.
- Key annotations: @StartSaga creates a new saga instance per trigger event.
- @SagaEventHandler(associationProperty = "orderId") routes events to the right instance by orderId .

*Applies **graph-traversal-algorithms** to the decision above.*

```java
@Saga
public class CheckoutSaga {

 @Inject
 private transient CommandGateway commandGateway; // transient = not serialized with saga state

 private String orderId;
 private String reservationId;
 private boolean paymentAuthorized;

 @StartSaga
 @SagaEventHandler(associationProperty = "orderId")
 public void on(OrderPlacedEvent event) {
 this.orderId = event.orderId();
 commandGateway.send(new ReserveInventoryCommand(event.orderId(), event.items()));
 }

 @SagaEventHandler(associationProperty = "orderId")
 public void on(InventoryReservedEvent event) {
 this.reservationId = event.reservationId();
 commandGateway.send(new AuthorizePaymentCommand(event.orderId(), event.totalCents()));
 }

 @SagaEventHandler(associationProperty = "orderId")
 public void on(PaymentAuthorizedEvent event) {
 this.paymentAuthorized = true;
 commandGateway.send(new CreateShipmentCommand(event.orderId()));
 SagaLifecycle.end();
 }

 // Compensation: inventory step failed — nothing reserved yet, just cancel the order
 @SagaEventHandler(associationProperty = "orderId")
 public void on(InventoryReservationFailedEvent event) {
 commandGateway.send(new CancelOrderCommand(event.orderId(), "inventory_unavailable"));
 SagaLifecycle.end();
 }

 // Compensation: payment failed — release reservation first, then cancel order
 @SagaEventHandler(associationProperty = "orderId")
 public void on(PaymentFailedEvent event) {
 commandGateway.send(new ReleaseInventoryCommand(orderId, reservationId));
 commandGateway.send(new CancelOrderCommand(orderId, "payment_failed"));
 SagaLifecycle.end();
 }
}
```

```mermaid
sequenceDiagram
 participant O as Orchestrator
 participant A as InventoryService
 participant B as PaymentService
 participant C as ShipmentService
 O->>A: ReserveInventory
 A-->>O: InventoryReserved
 O->>B: AuthorizePayment
 B-->>O: PaymentFailed
 O->>A: ReleaseInventory
 A-->>O: InventoryReleased
 Note over O: Saga compensated
```

**Observable outcome:** @SagaEventHandler(associationProperty = "orderId") routes events to the right instance by orderId .

## Stage 3 — Choreography in Code: Spring Kafka Payment Participant

- In the choreography style there is no central saga class.
- Each service subscribes to events and publishes its own outcome.
- The workflow emerges from the chain of reactions across services.
- The inventory service similarly listens on the payment-failed topic and releases its reservation.

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

```java
@Component
public class PaymentSagaParticipant {

 private final KafkaTemplate<String, Object> kafka;

 public PaymentSagaParticipant(KafkaTemplate<String, Object> kafka) {
 this.kafka = kafka;
 }

 @KafkaListener(topics = "order-placed")
 public void onOrderPlaced(OrderPlacedEvent event) {
 try {
 paymentService.authorize(event.orderId(), event.totalCents());
 kafka.send("payment-authorized", new PaymentAuthorizedEvent(event.orderId()));
 } catch (PaymentDeclinedException e) {
 kafka.send("payment-failed",
 new PaymentFailedEvent(event.orderId(), e.getReason()));
 }
 }
}
```

```mermaid
sequenceDiagram
 participant A as OrderService
 participant B as InventoryService
 participant C as PaymentService
 A->>B: OrderPlaced event
 B->>C: InventoryReserved event
 C->>A: PaymentFailed event
 A->>B: OrderCancelled event
 B-->>A: InventoryReleased event
 Note over A,C: Compensation via events
```

**Observable outcome:** The inventory service similarly listens on the payment-failed topic and releases its reservation.

## Stage 4 — Real-World Applications

- Sagas appear wherever a multi-step business process spans service boundaries: - Travel booking: flight → hotel → car rental.
- If the car is unavailable, compensate by cancelling hotel and flight in reverse order.
- - Bank money transfer: debit source → credit destination → notify parties.
- Compensation reverses the debit if the credit write fails.

*Applies **message-queues** to the decision above.*

**Observable outcome:** Compensation reverses the debit if the credit write fails.

## Stage 5 — Operator Field Note: Stuck Sagas and Dead Letters

- Alert on saga active duration — a checkout saga open for more than five minutes is almost certainly stuck.
- Axon routes commands that exhaust retries to a Dead Letter Queue (DLQ) ; inspect the entry with its orderId , replay if the failure was transient, or escalate for manual data correction.

*Applies **event-driven-architecture** to the decision above.*

**Observable outcome:** Axon routes commands that exhaust retries to a Dead Letter Queue (DLQ) ; inspect the entry with its orderId , replay if the failure was transient, or escalate for manual data correction.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - Design compensations before happy paths.
- If you cannot name the compensating transaction for each step at design time, the saga is not production-ready.

## Alternatives considered

### Decision Guide: When to Reach for Orchestration vs Choreography

Decision Guide: When to Reach for Orchestration vs Choreography


## Failure and degraded modes

### Compensating Transactions Are Not Rollbacks

- **Detection:** A compensating transaction is a new, forward-moving transaction that reverses the business effect of a prior step — it is not a database rollback.
- **User or operator impact:** A compensating transaction is a new, forward-moving transaction that reverses the business effect of a prior step — it is not a database rollback.

### Performance Analysis

- **Detection:** Sagas trade atomic rollback for latency decomposition .
- **User or operator impact:** Sagas trade atomic rollback for latency decomposition .

### Trade-offs & Failure Modes: Saga Trade-offs and the Failure Modes That Bite First

- **Detection:** No architecture pattern is free.
- **User or operator impact:** No architecture pattern is free.

## Scenario context

Any service that writes to a database **and** publishes a message in the same logical operation has a dual write problem. `try/catch` retries don't fix it — they turn failures into duplicates. The Transactional Outbox pattern co-writes business data and a pending event into the same DB transaction, then a relay delivers to Kafka separately. At-least-once delivery, idempotent consumers — no ghost orders, no phantom f…

This walkthrough focuses on **Why Every Instinctive Fix Makes the Problem Worse**.

Before making a design choice, clarify:

- When engineers first encounter the dual write problem, there are three instinctive fixes.
- All three are wrong — not because of implementation details, but because of structural incompatibilities with the systems involved.
- Fix Attempt 1 — "Just Wrap It in a Try-Catch and Retry" The problem is that the catch block cannot distinguish which write failed.

**Concepts in play:** event-driven-architecture, caching, api-gateway, transaction-isolation-distributed-acid

## Stage 1 — The Order That Silently Disappeared From Fulfillment

- It is 11:42 PM on Black Friday.
- An order service receives a checkout request, writes the new row to PostgreSQL — status PENDING , payment captured — and then calls kafkaTemplate.send("order-events", payload) .
- The Kafka broker is under peak load.
- A NetworkException is thrown, caught by catch (Exception e) { log.error(...); } , and silently swallowed.

*Applies **event-driven-architecture** to the decision above.*

**Observable outcome:** A NetworkException is thrown, caught by catch (Exception e) { log.error(...); } , and silently swallowed.

## Stage 2 — Deep Dive: How the Transactional Outbox Achieves True Atomicity

- The Transactional Outbox pattern resolves the dual write problem by changing the question.
- Instead of "how do we atomically write to DB and Kafka?" — which is impossible — it asks: "what if both writes go to the database, and we let a separate process deliver to Kafka?" The answer is that you can write the bu…
- The two rows either commit together or roll back together.
- Kafka delivery becomes a separate, retryable, eventually-consistent operation.

*Applies **caching** to the decision above.*

**Observable outcome:** Kafka delivery becomes a separate, retryable, eventually-consistent operation.

## Stage 3 — Visualizing the Dual Write Gap and the Outbox Solution End to End

- The diagram below contrasts the naive dual write (left path, failure gap visible) with the Transactional Outbox (right path, atomicity guaranteed through the shared DB transaction).
- The two subgraphs make the structural difference explicit.
- In the naive path, the commit and the Kafka send are sequential and independent — any exception between them creates a permanent gap.
- In the Outbox path, both writes happen inside one database transaction.

*Applies **api-gateway** to the decision above.*

```mermaid
flowchart TD
    subgraph naive[Naive Dual Write - failure gap]
        N1[orderRepo.save] --> N2[COMMIT success]
        N2 --> N3[kafkaTemplate.send]
        N3 -->|NetworkException| N4[Event lost - gap opens]
    end

    subgraph outbox[Transactional Outbox - atomic]
        O1[BEGIN TX] --> O2[INSERT INTO orders]
        O2 --> O3[INSERT INTO outbox_events published=false]
        O3 --> O4[COMMIT - both rows durable]
        O4 --> O5[Outbox Relay - Scheduled 500ms]
        O5 --> O6[KafkaTemplate.send]
        O6 -->|success| O7[UPDATE outbox_events published=true]
        O6 -->|failure| O8[Row stays unpublished - retried next cycle]
    end
```

**Observable outcome:** In the Outbox path, both writes happen inside one database transaction.

## Stage 4 — Where Dual Writes Appear Across Real Production Architecture Layers

- The dual write problem appears at every layer of a production system.
- Once you know the pattern, you see it everywhere.
- Order service (DB + Kafka): The canonical case.
- Save order → publish OrderPlaced .

*Applies **transaction-isolation-distributed-acid** to the decision above.*

```architecture
{
  "version": 1,
  "title": "Where Dual Writes Appear Across Real Production Architecture Layers - Subsystem Overview",
  "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.",
  "view": "high-level",
  "pattern": "microservices",
  "direction": "LR",
  "groups": [
    { "id": "edge", "label": "Edge Traffic Ingress", "tone": "slate" },
    { "id": "services", "label": "Stateless Microservices", "tone": "blue" },
    { "id": "data", "label": "Datastore Infrastructure", "tone": "green" }
  ],
  "nodes": [
    { "id": "gw", "component": "api-gateway", "label": "API Gateway", "group": "edge", "badge": "TLS & Auth" },
    { "id": "app", "component": "workflow-service", "label": "Domain Service", "group": "services", "badge": "Stateless" },
    { "id": "cache", "component": "redis", "label": "Redis Cache", "group": "data", "badge": "Sub-ms Reads" },
    { "id": "db", "component": "postgres", "label": "Transactional DB", "group": "data", "badge": "ACID Log" }
  ],
  "edges": [
    { "from": "gw", "to": "app", "kind": "request", "label": "gRPC Request" },
    { "from": "app", "to": "cache", "kind": "read", "label": "Cache Lookups" },
    { "from": "app", "to": "db", "kind": "write", "label": "Durable Write" }
  ],
  "takeaway": {
    "title": "Architecture Boundary Invariant",
    "body": "Stateless compute nodes scale horizontally while shielding underlying database clusters behind fast cache layers."
  }
}
```

**Observable outcome:** Save order → publish OrderPlaced .

## Stage 5 — Choosing the Right Pattern for Your System's Constraints

- The decision tree below maps system constraints to the appropriate pattern.
- Start with the type of secondary system you are writing to, then apply the latency and complexity budget.
- The flowchart above guides the initial pattern selection.
- Use the decision table below to validate your choice against operational constraints.

*Applies **event-driven-architecture** to the decision above.*

```mermaid
flowchart TD
    A[Service writes to two systems] --> B{Secondary system type?}
    B -->|Message broker Kafka SQS RabbitMQ| C{Sub-100ms latency required?}
    C -->|Yes| D[CDC + Debezium Outbox Router - 50ms WAL-based]
    C -->|No| E[Transactional Outbox Polling Relay - 500ms to 2s]
    B -->|Cache Redis Memcached| F[Cache-Aside DELETE on write - read-through repopulates]
    B -->|Search Index Elasticsearch| G[CDC to ES Sink Connector]
    B -->|Another service DB or cross-service state| H{Audit trail or time-travel required?}
    H -->|Yes| I[Event Sourcing + CQRS]
    H -->|No| J[Transactional Outbox + Saga for compensation]
```

**Observable outcome:** Use the decision table below to validate your choice against operational constraints.

## Stage 6 — Full Transactional Outbox Implementation in Spring Boot

- The following is a complete, production-ready implementation using Spring Data JPA for the database writes and Spring Kafka for event delivery.
- Each class maps directly to one of the three components described in the Deep Dive section.
- The OutboxEvent Entity — Staging Table for Pending Events The OrderService — Atomic Dual Write in a Single Transaction Both the business record and the outbox row are written inside one @Transactional boundary.
- If any step throws before the transaction commits, both rows roll back atomically.

*Applies **caching** to the decision above.*

```java
@Entity
@Table(name = "outbox_events", indexes = {
    @Index(name = "idx_outbox_pending", columnList = "published, created_at")
})
public class OutboxEvent {

    @Id
    private String eventId;

    private String aggregateType;   // "Order", "Payment", "User"
    private String aggregateId;     // business entity ID — becomes Kafka message key
    private String eventType;       // "OrderPlaced", "PaymentCaptured"

    @Column(columnDefinition = "TEXT")
    private String payload;         // JSON-serialized domain event

    private Instant createdAt;
    private boolean published;

    protected OutboxEvent() {}

    public OutboxEvent(String eventId, String aggregateType, String aggregateId,
                       String eventType, String payload, Instant createdAt, boolean published) {
        this.eventId = eventId;
        this.aggregateType = aggregateType;
        this.aggregateId = aggregateId;
        this.eventType = eventType;
        this.payload = payload;
        this.createdAt = createdAt;
        this.published = published;
    }

    public String getEventId()     { return eventId; }
    public String getAggregateId() { return aggregateId; }
    public String getPayload()     { return payload; }
    public boolean isPublished()   { return published; }
    public void setPublished(boolean p) { this.published = p; }
}
```

```java
@Service
@Transactional
public class OrderService {

    private final OrderRepository orderRepo;
    private final OutboxEventRepository outboxRepo;
    private final ObjectMapper mapper;

    public OrderService(OrderRepository orderRepo,
                        OutboxEventRepository outboxRepo,
                        ObjectMapper mapper) {
        this.orderRepo  = orderRepo;
        this.outboxRepo = outboxRepo;
        this.mapper     = mapper;
    }

    public Order placeOrder(PlaceOrderRequest req) throws JsonProcessingException {
        // Step 1: save business entity
        Order order = new Order(
            req.getCustomerId(),
            req.getItems(),
            OrderStatus.PENDING
        );
        orderRepo.save(order);

        // Step 2: write outbox event — SAME transaction as step 1
        String payload = mapper.writeValueAsString(
            new OrderPlacedEvent(order.getId(), req.getCustomerId(), order.getTotalAmount())
        );
        OutboxEvent event = new OutboxEvent(
            UUID.randomUUID().toString(),   // unique event_id — Kafka key for dedup
            "Order",
            order.getId().toString(),
            "OrderPlaced",
            payload,
            Instant.now(),
            false
        );
        outboxRepo.save(event);

        return order;
        // @Transactional commits both rows here — or rolls both back on any exception
    }
}
```

**Observable outcome:** If any step throws before the transaction commits, both rows roll back atomically.

## End-to-end outcome

When engineers first encounter the dual write problem, there are three instinctive fixes.

## Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

## Failure and degraded modes

### Before exploring solutions, it is essential to see all three failure modes. Engineers who only know Scenario A are surprised by B and C in production.

- **Detection:** Before exploring solutions, it is essential to see all three failure modes.
- **User or operator impact:** Before exploring solutions, it is essential to see all three failure modes.

### Scenario A — The Ghost Order: DB Succeeds, Kafka Fails

- **Detection:** The PostgreSQL transaction commits successfully.
- **User or operator impact:** The PostgreSQL transaction commits successfully.

### Scenario B — The Phantom Fulfillment: Kafka Succeeds, DB Fails

- **Detection:** The failure can run in the opposite direction, and the consequences are worse.
- **User or operator impact:** The failure can run in the opposite direction, and the consequences are worse.

### Scenario C — The Stale Cache: DB Succeeds, Redis Fails

- **Detection:** Not every dual write involves Kafka.
- **User or operator impact:** Not every dual write involves Kafka.

### Trade-offs and Failure Modes Across Outbox, CDC, and Event Sourcing

- **Detection:** No solution is universally optimal.
- **User or operator impact:** No solution is universally optimal.

### Hard Lessons from Dual Write Failures in Production

- **Detection:** Every service that writes to DB and publishes an event has a dual write problem.
- **User or operator impact:** Every service that writes to DB and publishes an event has a dual write problem.



Handle read/write separation, auditability, and multi-write failure modes.

Data and Consistency Patterns

## Scenario context

Circuit breakers protect callers from repeatedly hitting a failing dependency. They turn slow failure into fast failure, giving the rest of the system room to recover.



```architecture
{
 "ok": true,
 "diagram": {
 "version": 1,
 "title": "Circuit Breaker Pattern",
 "description": "Preventing cascading failures in service calls using circuit breakers",
 "view": "data-flow",
 "pattern": "microservi…

This walkthrough focuses on **When Circuit Breakers Actually Help**.

Before making a design choice, clarify:

- Use a circuit breaker when dependency failure can consume caller capacity faster than the dependency can recover.
- Strong fit cases: - user-facing APIs calling a flaky downstream service, - gateways or aggregators with many outbound calls, - paths where fallback or degraded behavior is acceptable, - systems where dependency timeouts…

**Concepts in play:** circuit-breaker, caching, canary-deployments, load-balancing

## Stage 1 — How a Breaker Works in Production

- The mechanics are simple but need disciplined thresholds: 1.
- Calls succeed in the closed state.
- If failures or slow calls cross the configured threshold, the breaker opens .
- In the open state, calls fail fast or use a fallback.

*Applies **circuit-breaker** to the decision above.*

**Observable outcome:** In the open state, calls fail fast or use a fallback.

## Stage 2 — Deep Dive: What Breaks First When Breakers Are Misused

- Field note: the most common operational mistake is setting a breaker and forgetting to alert on open state duration.
- If the breaker is open for twenty minutes and nobody notices, it protected capacity but still masked a user-visible outage.
- Internals: How Resilience4j Maintains State Resilience4j implements two sliding window strategies, selected via slidingWindowType : COUNT BASED stores the outcomes of the last N calls in a circular ring buffer of fixed-…
- Each new call result overwrites the oldest slot.

*Applies **caching** to the decision above.*

**Observable outcome:** Each new call result overwrites the oldest slot.

## Stage 3 — Circuit Breaker Flow

- This flowchart traces the full circuit breaker request lifecycle through all three breaker states — Closed, Open, and Half-Open.
- In the Closed state, requests reach the downstream dependency and failures accumulate in a sliding window until the threshold is crossed, tripping the breaker to Open; in the Open state all requests fast-fail to a fallb…
- The key takeaway is that the circuit breaker is a self-healing mechanism: it protects the dependency during recovery without requiring any operator intervention to reset.
- 📊 State Machine: CLOSED → OPEN → HALF OPEN This state diagram formalizes the three-state circuit breaker lifecycle as a finite state machine with precisely defined transition triggers.

*Applies **canary-deployments** to the decision above.*

```mermaid
flowchart TD
 A[Request needs downstream dependency] --> B{Breaker state}
 B -->|Closed| C[Call dependency]
 C --> D{Success or within timeout budget?}
 D -->|Yes| E[Return normal response]
 D -->|No| F[Record failure or slow-call event]
 F --> G{Threshold exceeded?}
 G -->|No| H[Keep breaker closed]
 G -->|Yes| I[Open breaker]
 B -->|Open| J[Fail fast or execute fallback]
 I --> K[Wait interval]
 K --> L[Half-open probe calls]
 L --> M{Probe succeeds?}
 M -->|Yes| N[Close breaker]
 M -->|No| I
```

```mermaid
stateDiagram-v2
 [*] --> Closed
 Closed --> Open : failure rate exceeds threshold
 Open --> HalfOpen : wait interval elapses
 HalfOpen --> Closed : all probe calls succeed
 HalfOpen --> Open : any probe call fails
 note right of Closed
 Normal traffic flows
 Records outcomes in window
 end note
 note right of Open
 Fail fast or fallback
 No calls to dependency
 end note
 note right of HalfOpen
 Limited probe calls
 Test dependency recovery
 end note
```

**Observable outcome:** 📊 State Machine: CLOSED → OPEN → HALF OPEN This state diagram formalizes the three-state circuit breaker lifecycle as a finite state machine with precisely defined transition triggers.

## Stage 4 — Concrete Config Example: Resilience4j Breaker Settings

- This Resilience4j circuit breaker YAML configuration targets a fraudService instance — a natural circuit breaker candidate because fraud detection is a non-critical dependency whose failure should never block the primar…
- The slidingWindowSize , failureRateThreshold , and waitDurationInOpenState fields map directly to the three state transitions shown in the state machine diagram above.
- Read the config top to bottom: the sliding window and failure rate threshold control the Closed-to-Open transition, the wait duration controls the Open-to-HalfOpen transition, and permittedNumberOfCallsInHalfOpenState c…
- Why these fields matter: - minimumNumberOfCalls avoids tripping on tiny sample noise.

*Applies **load-balancing** to the decision above.*

```yaml
resilience4j:
 circuitbreaker:
 instances:
 fraudService:
 slidingWindowType: COUNT_BASED
 slidingWindowSize: 50
 minimumNumberOfCalls: 20
 failureRateThreshold: 50
 slowCallRateThreshold: 60
 slowCallDurationThreshold: 500ms
 waitDurationInOpenState: 30s
 permittedNumberOfCallsInHalfOpenState: 5
 automaticTransitionFromOpenToHalfOpenEnabled: true
```

**Observable outcome:** Why these fields matter: - minimumNumberOfCalls avoids tripping on tiny sample noise.

## Stage 5 — Spring Boot Implementation: Protecting a Fraud Service Call

- The YAML config above tells Resilience4j when to trip.
- The code below wires what happens when it does.
- The scenario: CheckoutService calls FraudService .
- When the fraud service is slow or erroring, the breaker trips and a fallback returns ALLOW WITH REVIEW — checkout stays operational with an audit log entry rather than surfacing a hard error to the customer.

*Applies **circuit-breaker** to the decision above.*

```xml
<dependency>
 <groupId>io.github.resilience4j</groupId>
 <artifactId>resilience4j-spring-boot3</artifactId>
 <version>2.2.0</version>
</dependency>
<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
 <groupId>io.micrometer</groupId>
 <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
```

```yaml
management:
 endpoints:
 web:
 exposure:
 include: health,metrics,circuitbreakers
 health:
 circuitbreakers:
 enabled: true
```

**Observable outcome:** When the fraud service is slow or erroring, the breaker trips and a fallback returns ALLOW WITH REVIEW — checkout stays operational with an audit log entry rather than surfacing a hard error to the customer.

## Stage 6 — Real-World Applications: What to Instrument and What to Alert On

- What breaks first in production: 1.
- Slow calls that do not count as failures.
- Fallback paths that were never load-tested.
- Alerting that focuses on 5xx only and misses degraded open-state behavior.

*Applies **caching** to the decision above.*

**Observable outcome:** Alerting that focuses on 5xx only and misses degraded open-state behavior.

## End-to-end outcome

Use a circuit breaker when dependency failure can consume caller capacity faster than the dependency can recover.

## Alternatives considered

### Decision Guide for Dependency Protection

If your service cannot explain what users receive when the breaker is open, the design is incomplete.


## Failure and degraded modes

### When Not to Use Circuit Breakers

- **Detection:** Breakers are not a substitute for basic dependency hygiene.
- **User or operator impact:** Breakers are not a substitute for basic dependency hygiene.

### Trade-offs & Failure Modes: Pros, Cons, and Alternatives

- **Detection:** Trade-offs & Failure Modes: Pros, Cons, and Alternatives
- **User or operator impact:** Trade-offs & Failure Modes: Pros, Cons, and Alternatives

## Scenario context

Bulkheads isolate capacity so one overloaded dependency or workload class cannot consume every thread, queue slot, or connection in the service.



```architecture
{
 "ok": true,
 "diagram": {
 "version": 1,
 "title": "Bulkhead Pattern: Isolating Capacity",
 "description": "The system topology utilizes bulkheads to isolate capacity and protect critical workloads from overloaded dependencies or workload cl…

This walkthrough focuses on **When the Bulkhead Pattern Actually Helps**.

Before making a design choice, clarify:

- Bulkheads are useful when requests compete for shared runtime resources: thread pools, connection pools, worker queues, CPU quotas, or outbound concurrency budgets.
- Use bulkheads when: - one dependency is slower or riskier than the rest, - critical and best-effort traffic share the same service process, - noisy tenants or expensive operations can consume disproportionate capacity,…

**Concepts in play:** circuit-breaker, stack-and-queue-data-structures, api-gateway, database-indexing

## Stage 1 — How Bulkheads Work in Production

- Bulkheads are most effective when the isolation boundary matches the operational risk.
- Typical implementation sequence: 1.
- Identify the resource that actually starves first: threads, DB connections, queue workers, outbound sockets, or CPU.
- Split critical and non-critical paths into separate budgets.

*Applies **circuit-breaker** to the decision above.*

**Observable outcome:** Split critical and non-critical paths into separate budgets.

## Stage 2 — Deep Dive: What Incident Reviews Usually Reveal First

- The biggest mistakes are usually classification mistakes.
- Field note: bulkheads fail most often when teams isolate execution pools but forget shared downstream resources.
- If every pool still hits the same saturated connection pool, the isolation is cosmetic.
- Internals: Semaphore vs Thread Pool Isolation Contracts Resilience4j ships two structurally distinct bulkhead mechanisms — each enforces isolation in a fundamentally different way.

*Applies **stack-and-queue-data-structures** to the decision above.*

**Observable outcome:** Internals: Semaphore vs Thread Pool Isolation Contracts Resilience4j ships two structurally distinct bulkhead mechanisms — each enforces isolation in a fundamentally different way.

## Stage 3 — Bulkhead Runtime Flow

- This flowchart shows the bulkhead router classifying every incoming request by workload class and directing it to a dedicated pool with its own capacity budget.
- Critical requests are fast-failed with an alert if their budget is exhausted; best-effort requests are rejected, deferred, or served stale results when their budget runs out.
- The key insight is that pool exhaustion in one workload class has zero impact on the other — the blast radius of any overload event is bounded by design, not by luck.
- 📊 Thread Pool Partitioning: Failure Isolation This diagram shows three completely isolated thread pools — Checkout (40 permits), Reporting (8 permits), and Email (8 threads) — each serving its own downstream dependency.

*Applies **api-gateway** to the decision above.*

```mermaid
flowchart TD
 A[Incoming request] --> B{Workload class?}
 B -->|Critical| C[Critical pool and queue]
 B -->|Best effort| D[Best-effort pool and queue]
 C --> E[Protected dependency path]
 D --> F[Non-critical dependency path]
 D --> G{Best-effort budget exhausted?}
 G -->|Yes| H[Reject, defer, or serve stale result]
 G -->|No| F
 C --> I{Critical budget exhausted?}
 I -->|Yes| J[Fast fail and alert]
 I -->|No| E
```

```mermaid
flowchart TD
 subgraph Checkout Pool - 40 permits
 CP1[Request 1] --> CA[Payment Auth]
 CP2[Request 2] --> CA
 end
 subgraph Reporting Pool - 8 permits
 RP1[Report Job] --> RE[Export Service]
 RP2[Report Job] --> RE
 end
 subgraph Email Pool - 8 threads
 EP1[Email Task] --> ES[Email Service]
 end
 RE -->|Reporting pool fails| X[Pool Exhausted BulkheadFullException]
 X -->|isolated| CA
 CA -->|unaffected| OK[Checkout continues]
```

**Observable outcome:** 📊 Thread Pool Partitioning: Failure Isolation This diagram shows three completely isolated thread pools — Checkout (40 permits), Reporting (8 permits), and Email (8 threads) — each serving its own downstream dependency.

## Stage 4 — Concrete Config Example: Resilience4j Bulkhead Budgets

- Why this is useful operationally: - paymentAuth gets a higher protected budget than reporting.
- - maxWaitDuration: 0 avoids hidden queueing for interactive paths.
- - Separate thread-pool bulkheads make worker contention visible and tunable.

*Applies **database-indexing** to the decision above.*

```yaml
resilience4j:
 bulkhead:
 instances:
 paymentAuth:
 maxConcurrentCalls: 40
 maxWaitDuration: 0
 reportingExport:
 maxConcurrentCalls: 8
 maxWaitDuration: 0
 thread-pool-bulkhead:
 instances:
 emailFanout:
 coreThreadPoolSize: 4
 maxThreadPoolSize: 8
 queueCapacity: 200
 reconciliation:
 coreThreadPoolSize: 6
 maxThreadPoolSize: 12
 queueCapacity: 50
```

**Observable outcome:** - Separate thread-pool bulkheads make worker contention visible and tunable.

## Stage 5 — Real-World Applications: What to Instrument and What Breaks First

- Bulkheads are only valuable if you can see saturation early.
- Critical path still shares an unseen downstream bottleneck.
- Best-effort queue grows quietly until operators notice user impact.
- Capacity split is tuned once and never revisited.

*Applies **circuit-breaker** to the decision above.*

**Observable outcome:** Capacity split is tuned once and never revisited.

## Stage 6 — Resilience4j: Semaphore and Thread Pool Bulkheads for Spring Boot Services

- Resilience4j is a lightweight fault-tolerance library designed for Java 8+ and Spring Boot, providing semaphore and thread-pool bulkhead implementations as first-class Spring beans with Micrometer metrics integration an…
- How it solves the problem: The checkout-vs-reporting isolation design described throughout this post maps directly to Resilience4j's two bulkhead types.
- @Bulkhead(type = SEMAPHORE) protects synchronous user-facing paths with a hard concurrency cap and zero queue; @Bulkhead(type = THREADPOOL) isolates async background work on a dedicated executor, ensuring the servlet th…
- The full implementation — including the paymentAuth and reportingExport YAML configuration, CheckoutService , ReportingService , and Micrometer metric names — is covered in detail in the Spring Boot Implementation secti…

*Applies **stack-and-queue-data-structures** to the decision above.*

```xml
<dependency>
 <groupId>io.github.resilience4j</groupId>
 <artifactId>resilience4j-spring-boot3</artifactId>
 <version>2.2.0</version>
</dependency>
<dependency>
 <groupId>io.github.resilience4j</groupId>
 <artifactId>resilience4j-micrometer</artifactId>
 <version>2.2.0</version>
</dependency>
```

**Observable outcome:** The full implementation — including the paymentAuth and reportingExport YAML configuration, CheckoutService , ReportingService , and Micrometer metric names — is covered in detail in the Spring Boot Implementation secti…

## End-to-end outcome

Bulkheads are useful when requests compete for shared runtime resources: thread pools, connection pools, worker queues, CPU quotas, or outbound concurrency budgets.

## Alternatives considered

### Scenario**: `OrderController` serves checkout requests (critical, user-facing) and finance export requests (best-effort, async). Reporting must never compete with checkout for servlet threads.

Maven dependency (Spring Boot 3): The bulkhead namespace in the YAML ( paymentAuth , reportingExport ) is semaphore-based : calls run on the caller's thread with a hard concurrency cap and zero queue.

### Semaphore Bulkhead on the Checkout Path

Checkout is synchronous and user-facing.

### Thread Pool Bulkhead on the Reporting Path

Finance export runs on a dedicated thread pool separate from the servlet pool.

### Micrometer Metrics for Both Paths

Resilience4j emits bulkhead state to Micrometer automatically when resilience4j-micrometer is on the classpath: Alert when available concurrent calls{name="paymentAuth"} reaches 0 — that is the exact moment live checkouts begin seeing rejections.

### Decision Guide for Capacity Isolation

If you cannot say which resource is being protected, you do not yet have a bulkhead design.


## Failure and degraded modes

### When Not to Use Bulkheads

- **Detection:** Bulkheads add complexity and can waste capacity if you split resources without real contention patterns.
- **User or operator impact:** Bulkheads add complexity and can waste capacity if you split resources without real contention patterns.

### Trade-offs & Failure Modes: Pros, Cons, and Alternatives

- **Detection:** Trade-offs & Failure Modes: Pros, Cons, and Alternatives
- **User or operator impact:** Trade-offs & Failure Modes: Pros, Cons, and Alternatives

## Scenario context

A dead letter queue protects throughput by moving repeatedly failing messages out of the hot path. It only works if retries are bounded, triage has an owner, and replay is a deliberate workflow instead of a panic button.

> **TLDR**: The main SRE question is not “do we have a DLQ?” It is “what exactly lands there, who wakes up when it does, and how do we replay safely without looping the incident back into productio…

This walkthrough focuses on **When a DLQ Actually Helps**.

Before making a design choice, clarify:

- DLQs are useful when some messages must fail independently so the rest of the stream can keep moving.
- Use them when: - one poison message can block a partition or worker loop, - retryable and non-retryable failures need different handling, - operators need a durable place to inspect failed payloads, - the system must pr…

**Concepts in play:** message-queues, event-driven-architecture, stack-and-queue-data-structures, jvm-internals-and-garbage-collection

## Stage 1 — How DLQs Work in Production

- The healthy pattern is simple: 1.
- Main consumer processes messages.
- Transient failures retry with backoff and a hard cap.
- After retry exhaustion or explicit non-retryable classification, the message moves to the DLQ.

*Applies **message-queues** to the decision above.*

**Observable outcome:** After retry exhaustion or explicit non-retryable classification, the message moves to the DLQ.

## Stage 2 — Deep Dive: Incident Patterns in DLQ Systems

- Field note: the most dangerous replay command is the one that says “send everything back.” Good replay is scoped by failure cause, code version, and idempotency guarantees.
- Internals: How DeadLetterPublishingRecoverer Routes Failed Messages In Spring Kafka, the DefaultErrorHandler intercepts every exception that escapes a @KafkaListener .
- On each failure it applies the configured BackOff policy.
- Once the retry budget is exhausted — or the exception is classified as non-retryable — it delegates to a DeadLetterPublishingRecoverer , which publishes the original record verbatim to a dead-letter topic.

*Applies **event-driven-architecture** to the decision above.*

**Observable outcome:** Once the retry budget is exhausted — or the exception is classified as non-retryable — it delegates to a DeadLetterPublishingRecoverer , which publishes the original record verbatim to a dead-letter topic.

## Stage 3 — DLQ Runtime Flow

- This diagram shows the full runtime decision tree a consumer follows for every message: acknowledge on success, retry with backoff on transient failure, then route to the DLQ when the retry budget is exhausted or the er…
- The operator triage branch at the bottom makes explicit that DLQ messages do not automatically fix themselves — they require an explicit replay or discard decision.
- The key takeaway is that the DLQ is not a destination for bad messages; it is a quarantine that demands a human or automated resolution workflow.
- 📊 Poison Message Lifecycle: Producer to DLQ to Inspector This sequence diagram walks the complete lifecycle of a poison message from the producer's perspective through three failed delivery attempts, culminating in the…

*Applies **stack-and-queue-data-structures** to the decision above.*

```mermaid
flowchart TD
 A[Main queue message] --> B[Consumer attempts processing]
 B --> C{Success?}
 C -->|Yes| D[Ack and continue]
 C -->|No| E{Retryable?}
 E -->|Yes| F[Retry with backoff]
 F --> G{Retry budget exhausted?}
 G -->|No| B
 G -->|Yes| H[Move to DLQ with error metadata]
 E -->|No| H
 H --> I[Operator triage and fix]
 I --> J{Replay safe?}
 J -->|Yes| K[Scoped replay]
 J -->|No| L[Hold, purge, or corrective workflow]
```

```mermaid
sequenceDiagram
 participant P as Producer
 participant Q as Main Queue
 participant C as Consumer
 participant DLQ as Dead Letter Queue
 participant Ops as Inspector
 P->>Q: publish message
 Q->>C: deliver message (attempt 1)
 C-->>Q: NACK / processing failed
 Q->>C: redeliver (attempt 2)
 C-->>Q: NACK / processing failed
 Q->>C: redeliver (attempt 3)
 C-->>Q: NACK - retry budget exhausted
 Q->>DLQ: route with error headers
 DLQ->>Ops: alert on message age
 Ops->>DLQ: inspect payload and error
 Ops->>Q: scoped replay after fix
```

**Observable outcome:** 📊 Poison Message Lifecycle: Producer to DLQ to Inspector This sequence diagram walks the complete lifecycle of a poison message from the producer's perspective through three failed delivery attempts, culminating in the…

## Stage 4 — Concrete Config Example: SQS Redrive Policy

- For AWS SQS consumers, the broker-side equivalent uses a redrive policy — no application code required, but also no control over retry timing or per-exception routing: Why this matters operationally: - maxReceiveCount d…
- - deadLetterTargetArn makes the isolation path explicit.
- - MessageRetentionPeriod determines how long the team has to investigate and replay.

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

```json
{
 "RedrivePolicy": {
 "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789012:notifications-dlq",
 "maxReceiveCount": 5
 },
 "VisibilityTimeout": 60,
 "MessageRetentionPeriod": 1209600
}
```

**Observable outcome:** - MessageRetentionPeriod determines how long the team has to investigate and replay.

## Stage 5 — Real-World Applications

- 

*Applies **message-queues** to the decision above.*

**Observable outcome:** You can explain how “Real-World Applications” changes the architecture or implementation path.

## Stage 6 — Spring for Apache Kafka and AWS SQS DLQ: Dead-Letter Delivery in Practice

- Spring for Apache Kafka is the Spring integration library that wraps the Kafka client with @KafkaListener , KafkaTemplate , DefaultErrorHandler , and DeadLetterPublishingRecoverer — giving Spring Boot services productio…
- AWS SQS with a redrive policy is the managed-broker equivalent: the queue broker itself enforces the retry cap and routes exhausted messages to the DLQ without any application code.
- Spring for Apache Kafka solves the DLQ problem by intercepting exceptions that escape @KafkaListener , applying a configurable BackOff policy, and — on retry exhaustion — publishing the original record to a .DLT topic w…
- For AWS SQS consumers, the broker-enforced redrive policy ( maxReceiveCount + deadLetterTargetArn ) achieves the same isolation without application code — shown in the 🧪 Concrete Config Example section above.

*Applies **event-driven-architecture** to the decision above.*

```java
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.retry.annotation.Backoff;
import org.springframework.stereotype.Component;

@Component
public class OrderEventConsumer {

 /**
 * @RetryableTopic configures retries and DLT routing declaratively.
 * Spring creates topic "orders.retry-0", "orders.retry-1", "orders.DLT"
 * automatically; no manual topic provisioning needed.
 */
 @RetryableTopic(
 attempts = "4", // 1 original + 3 retries
 backoff = @Backoff(delay = 1000, multiplier = 2.0, maxDelay = 30000),
 dltTopicSuffix = ".DLT",
 include = {TransientProcessingException.class} // only retry transient failures
 )
 @KafkaListener(topics = "orders", groupId = "order-processor")
 public void processOrder(ConsumerRecord<String, OrderEvent> record) {
 OrderEvent event = record.value();
 // Throw TransientProcessingException for retryable failures.
 // Non-retryable exceptions skip retries and route straight to the DLT.
 orderService.process(event);
 }

 // Separate listener on the DLT for triage and alerting
 @KafkaListener(topics = "orders.DLT", groupId = "order-dlq-triage")
 public void handleDeadLetter(ConsumerRecord<String, OrderEvent> record) {
 String originalTopic = new String(record.headers()
 .lastHeader("kafka_dlt-original-topic").value());
 String exceptionMessage = new String(record.headers()
 .lastHeader("kafka_dlt-exception-message").value());

 dlqAlertService.notify(record.key(), originalTopic, exceptionMessage);
 dlqRepository.save(new DlqEntry(record.key(), originalTopic, exceptionMessage));
 }
}
```

**Observable outcome:** For AWS SQS consumers, the broker-enforced redrive policy ( maxReceiveCount + deadLetterTargetArn ) achieves the same isolation without application code — shown in the 🧪 Concrete Config Example section above.

## End-to-end outcome

DLQs are useful when some messages must fail independently so the rest of the stream can keep moving.

## Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

## Failure and degraded modes

### When Not to Use a DLQ

- **Detection:** DLQs are not a substitute for proper retry classification or owning the underlying bug.
- **User or operator impact:** DLQs are not a substitute for proper retry classification or owning the underlying bug.

### Trade-offs & Failure Modes: Pros, Cons, and Alternatives

- **Detection:** Trade-offs & Failure Modes: Pros, Cons, and Alternatives
- **User or operator impact:** Trade-offs & Failure Modes: Pros, Cons, and Alternatives

### Decision Guide for Failure Isolation

- **Detection:** If the team cannot answer who owns the oldest DLQ message, the design is incomplete.
- **User or operator impact:** If the team cannot answer who owns the oldest DLQ message, the design is incomplete.

## Scenario context

As microservice architectures scale, a single outage in a core service can cascade across the entire system. Cell-Based Architecture mitigates this by partitioning the entire system into small, self-contained, independent units called cells. This guide outlines their routing mechanics and provides a custom Spring Cloud Gateway filter for cell-based traffic distribution.






### The Partitioning Key
To assign users…

This walkthrough focuses on **Flow: Cell Routing Sequence**.

Before making a design choice, clarify:

- The diagram below maps the execution path of a user request intercepted by the smart gateway router and dispatched to isolated cell clusters: The table below contrasts the layout properties of a traditional global micro…

**Concepts in play:** jvm-internals-and-garbage-collection, database-sharding, database-indexing, caching

## Stage 1 — Deep Dive: Dynamic Routing and Blast-Radius Control

- Implementing cell-based routing requires a robust gateway layer to handle request parsing, tenant routing lookup, and latency overhead constraints.
- Gateway Routing Filter Internals The Cell Router Gateway acts as a reverse proxy.
- It parses incoming HTTP request metadata, checks the route lookup mapping, and updates the target routing URI dynamically.
- Because the gateway is the entry point for all traffic, it must be stateless, highly concurrent, and non-blocking.

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

**Observable outcome:** Because the gateway is the entry point for all traffic, it must be stateless, highly concurrent, and non-blocking.

## Stage 2 — Advanced Concepts: Multi-Cell Data Replication and Failover

- While cells are isolated, some operations require cross-cell data access.
- Gateway Routing Configuration (YAML) This configuration defines the gateway routes, directing traffic to a custom CellRoutingFilter to resolve target destinations.
- Spring Cloud Gateway CellRoutingFilter Code This Java class implements a gateway filter that extracts the X-Tenant-ID header, determines the target cell destination, and overwrites the routing URI dynamically.

*Applies **database-sharding** to the decision above.*

```yaml
spring:
 cloud:
 gateway:
 routes:
 - id: cell-route-wildcard
 uri: http://fallback-cell-service
 predicates:
 - Path=/api/v1/**
 filters:
 - name: CellRoutingFilter
```

```java
import org.springframework.cloud.gateway.filter.GatewayFilter;
import org.springframework.cloud.gateway.filter.GatewayFilterChain;
import org.springframework.cloud.gateway.filter.factory.AbstractGatewayFilterFactory;
import org.springframework.cloud.gateway.support.ServerWebExchangeUtils;
import org.springframework.http.server.reactive.ServerHttpRequest;
import org.springframework.stereotype.Component;
import org.springframework.web.server.ServerWebExchange;
import reactor.core.publisher.Mono;

import java.net.URI;
import java.util.HashMap;
import java.util.Map;

@Component
public class CellRoutingFilter extends AbstractGatewayFilterFactory<CellRoutingFilter.Config> {

 // Simulating an in-memory routing directory mapping tenant IDs to cell URIs
 private static final Map<String, String> CELL_REGISTRY = new HashMap<>();

 static {
 CELL_REGISTRY.put("T101", "http://cell1-service.local:8081");
 CELL_REGISTRY.put("T102", "http://cell2-service.local:8082");
 CELL_REGISTRY.put("T103", "http://cell3-service.local:8083");
 }

 public CellRoutingFilter() {
 super(Config.class);
 }

 @Override
 public GatewayFilter apply(Config config) {
 return (exchange, chain) -> {
 ServerHttpRequest request = exchange.getRequest();
 
 // Extract the partition key (tenant_id) from the headers
 String tenantId = request.getHeaders().getFirst("X-Tenant-ID");
 
 if (tenantId == null || tenantId.isEmpty()) {
 // Return 400 Bad Request if partition key is missing
 exchange.getResponse().setStatusCode(org.springframework.http.HttpStatus.BAD_REQUEST);
 return exchange.getResponse().setComplete();
 }

 // Resolve target cell URI using the registry, fallback if not found
 String targetCellUri = CELL_REGISTRY.getOrDefault(tenantId, "http://cell-fallback-service.local:8080");

 // Build new destination URI
 URI newUri = URI.create(targetCellUri + request.getPath().value());

 // Overwrite the GATEWAY_REQUEST_URL_ATTR with our new cell URI
 exchange.getAttributes().put(ServerWebExchangeUtils.GATEWAY_REQUEST_URL_ATTR, newUri);

 System.out.println(String.format("CellRouter: Routing tenant '%s' request to cell '%s'", tenantId, newUri));

 return chain.filter(exchange);
 };
 }

 public static class Config {
 // Configuration parameters can be defined here
 }
}
```

**Observable outcome:** Spring Cloud Gateway CellRoutingFilter Code This Java class implements a gateway filter that extracts the X-Tenant-ID header, determines the target cell destination, and overwrites the routing URI dynamically.

## End-to-end outcome

After working through the stages, you should be able to explain:

- Allowing Cross-Cell Database Joins: Connecting one cell's database to another cell's service breaks isolation.
- If the target cell database goes down, both cells will fail.
- Cell: An independent, self-contained deployment unit containing application servers, databases, and caches.
- Smart Gateway: Intercepts traffic at the network edge and routes requests to the correct cell based on a partitioning key.

## Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

## Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.

Build systems that degrade instead of collapsing.

Resilience Patterns

## Scenario context

Large-scale modernization usually fails when teams try to replace an entire legacy platform in one synchronized rewrite. The safer approach is to create seams, translate old contracts into stable new ones, and move traffic gradually with measurable rollback points.

> **TLDR**: Modernization is safe only when it is reversible: build seams first, translate semantics explicitly, and migrate traffic in measurable steps…

This walkthrough focuses on **Real-World Applications: Realistic Scenario: Billing Extraction from Legacy Commerce Core**.

Before making a design choice, clarify:

- Constraints: - Legacy monolith handles billing, checkout, and support adjustments.
- - Refund correctness must exceed 99.95%.
- - Cutover window cannot exceed 10 minutes.

**Concepts in play:** design-patterns-gof, api-gateway, sorting-and-searching-algorithms, consistency-replication

## Stage 1 — When Modernization Patterns Beat Big-Bang Rewrites

- Legacy systems survive because they contain real business knowledge.
- Rewriting everything at once usually loses hidden rules and creates high-risk cutovers.
- Use modernization patterns to change architecture without breaking meaning.

*Applies **design-patterns-gof** to the decision above.*

**Observable outcome:** Use modernization patterns to change architecture without breaking meaning.

## Stage 2 — When to Use Strangler Fig, ACL, and Modular Monolith

- When not to use these patterns - If the system is tiny and low-risk, full rewrite might be cheaper.
- - If you cannot define rollback points, do not begin migration traffic moves.

*Applies **api-gateway** to the decision above.*

**Observable outcome:** - If you cannot define rollback points, do not begin migration traffic moves.

## Stage 3 — How Seam-First Modernization Works

- Choose one business capability with clear owner.
- Define stable contract for callers.
- Insert seam (gateway/facade/abstraction).
- Add ACL to isolate target model from legacy semantics.

*Applies **sorting-and-searching-algorithms** to the decision above.*

**Observable outcome:** Add ACL to isolate target model from legacy semantics.

## Stage 4 — How to Implement: Migration Checklist

- Baseline current behavior and edge-case inventory.
- Build seam at API/gateway/facade boundary.
- Create ACL mapping table for entities and enums.
- Add shadow reads or mirrored execution for comparison.

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** Add shadow reads or mirrored execution for comparison.

## Stage 5 — Deep Dive: Internals of Safe Extraction

- The Internals: Routing Seams, Translation Rules, and Dual-Run Safety Seam design determines migration quality more than the new service framework.
- Core seam components: - routing decision point, - stable caller contract, - ACL translation layer, - comparison and audit telemetry.
- ACL best practice: make translation tables explicit and versioned.
- Do not hide semantic mappings inside ad hoc code branches.

*Applies **design-patterns-gof** to the decision above.*

```mermaid
sequenceDiagram
    participant NS as NewService
    participant ACL as ACL Translator
    participant LS as LegacySystem
    NS->>ACL: Call in new domain model
    ACL->>ACL: Translate to legacy model
    ACL->>LS: Call legacy API
    LS-->>ACL: Legacy response
    ACL->>ACL: Translate to new model
    ACL-->>NS: Clean domain response
    Note over ACL: Shields new service
```

**Observable outcome:** Do not hide semantic mappings inside ad hoc code branches.

## Stage 6 — Strangler Migration Flow: Route, Translate, Compare, Promote

- This flowchart maps the complete request journey through a live Strangler Fig migration: the seam gateway decides per-capability whether to route to the legacy path or through the ACL to the new service, then comparison…
- The divergence check at the bottom is what makes continuous safe traffic promotion possible—when outputs align within the agreed threshold, migrated traffic increases; when they diverge, an automatic rollback is trigger…
- The seam gateway is the single most important control point in the entire migration because all rollback and promotion decisions flow through it.

*Applies **api-gateway** to the decision above.*

```mermaid
flowchart TD
  A[Client request] --> B[Seam gateway or facade]
  B --> C{Migrated capability?}
  C -->|No| D[Legacy path]
  C -->|Yes| E[ACL translation]
  E --> F[New service path]
  D --> G[Legacy datastore]
  F --> H[Target datastore]
  D --> I[Comparison telemetry]
  F --> I
  I --> J{Divergence within threshold?}
  J -->|Yes| K[Increase migrated traffic]
  J -->|No| L[Rollback and investigate]
```

**Observable outcome:** The seam gateway is the single most important control point in the entire migration because all rollback and promotion decisions flow through it.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - Seam quality is more important than migration speed.
- - ACLs prevent legacy semantics from polluting new domain boundaries.

## Alternatives considered

### Decision Guide: First Pattern to Apply

If rollback route is unclear, postpone migration traffic expansion.


## Failure and degraded modes

### Trade-offs & Failure Modes: Pros, Cons, and Risks in Modernization Programs

- **Detection:** Trade-offs & Failure Modes: Pros, Cons, and Risks in Modernization Programs
- **User or operator impact:** Trade-offs & Failure Modes: Pros, Cons, and Risks in Modernization Programs



## Scenario context

Altering database schemas in a high-traffic production system without downtime requires the Expand-Contract pattern. Instead of changing a column in place, we add a new column (Expand), double-write to both columns via the application (Transition), backfill historical data, and finally remove the old column (Contract). This guide details this lifecycle with full Spring Boot and Liquibase snippets.

* * *



```archi…

This walkthrough focuses on **Concept: Zero-Downtime Schema Migrations**.

Before making a design choice, clarify:

- In modern distributed systems, deploying updates to application servers is routinely managed using rolling updates or blue-green deployments.
- However, databases present a unique challenge because state cannot be easily duplicated or discarded.
- Running a standard ALTER TABLE RENAME COLUMN statement on a table containing millions of rows requires acquiring an exclusive table lock, causing active queries to block, requests to time out, and leading to service out…

**Concepts in play:** caching, jvm-internals-and-garbage-collection, consistency-replication, api-gateway

## Stage 1 — Mechanics: The Three-Phase Migration Lifecycle

- Let us trace the evolution of a schema during a column rename operation.
- Suppose we want to rename the column phone number to contact phone in our users table.
- We execute the change through these structured phases: 1.
- Phase 1: Expand (Database): We add the new column contact phone to the database schema as a nullable field.

*Applies **caching** to the decision above.*

**Observable outcome:** Phase 1: Expand (Database): We add the new column contact phone to the database schema as a nullable field.

## Stage 2 — Flow: High-Level Migration Sequence

- The diagram below visualizes the transition states of the database and the active application versions during the migration: The table below summarizes the operational compatibility matrices at each stage:

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

```architecture
{
 "version": 1,
 "title": "Flow: High-Level Migration Sequence - Subsystem Overview",
 "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.",
 "view": "high-level",
 "pattern": "microservices",
 "direction": "LR",
 "groups": [
 { "id": "edge", "label": "Edge Traffic Ingress", "tone": "slate" },
 { "id": "services", "label": "Stateless Microservices", "tone": "blue" },
 { "id": "data", "label": "Datastore Infrastructure", "tone": "green" }
 ],
 "nodes": [
 { "id": "gw", "component": "api-gateway", "label": "API Gateway", "group": "edge", "badge": "TLS & Auth" },
 { "id": "app", "component": "workflow-service", "label": "Domain Service", "group": "services", "badge": "Stateless" },
 { "id": "cache", "component": "redis", "label": "Redis Cache", "group": "data", "badge": "Sub-ms Reads" },
 { "id": "db", "component": "postgres", "label": "Transactional DB", "group": "data", "badge": "ACID Log" }
 ],
 "edges": [
 { "from": "gw", "to": "app", "kind": "request", "label": "gRPC Request" },
 { "from": "app", "to": "cache", "kind": "read", "label": "Cache Lookups" },
 { "from": "app", "to": "db", "kind": "write", "label": "Durable Write" }
 ],
 "takeaway": {
 "title": "Architecture Boundary Invariant",
 "body": "Stateless compute nodes scale horizontally while shielding underlying database clusters behind fast cache layers."
 }
}
```

```mermaid
graph TD
 Start["State 0: Legacy Schema App V0 reads/writes 'phone_number'"] -->|Expand: Add Column| Phase1["State 1: Expanded Schema App V1 double-writes to both, reads 'phone_number'"]
 Phase1 -->|Data Backfill| Backfill["State 2: Backfilled Schema All historical values copied to 'contact_phone'"]
 Backfill -->|Read Switch| Phase2["State 3: Transition Schema App V2 double-writes to both, reads 'contact_phone'"]
 Phase2 -->|Contract: Drop Old Column| Phase3["State 4: Contracted Schema App V3 reads/writes 'contact_phone' only"]
```

**Observable outcome:** The diagram below visualizes the transition states of the database and the active application versions during the migration: The table below summarizes the operational compatibility matrices at each stage:

## Stage 3 — Deep Dive: Structural Evolution & Performance Profiling

- Understanding how the database engine executes these operations reveals why the Expand-Contract pattern is necessary.
- Database Engine Lock Internals In relational database engines like PostgreSQL or MySQL (InnoDB), schema changes require obtaining a Metadata Lock (MDL).
- In older database versions, any ALTER TABLE operation required rebuilding the entire table on disk, acquiring an Exclusive (Access Exclusive) lock.
- This blocks all read and write queries on the target table.

*Applies **consistency-replication** to the decision above.*

**Observable outcome:** This blocks all read and write queries on the target table.

## Stage 4 — Advanced Concepts: Handling Large-Scale Table Alterations

- When working with tables that exceed hundreds of gigabytes, even adding a nullable column can sometimes hit system boundaries (e.g., maximum row size limits or disk space limits during page splits).
- To handle these scenarios at scale, we use toolkits like GitHub's gh-ost or Percona's pt-online-schema-change .
- These tools create a shadow copy of the table, apply migrations to the shadow table, set up database triggers or parse binary logs to stream incoming writes from the live table to the shadow table, and then execute a fa…

*Applies **api-gateway** to the decision above.*

**Observable outcome:** These tools create a shadow copy of the table, apply migrations to the shadow table, set up database triggers or parse binary logs to stream incoming writes from the live table to the shadow table, and then execute a fa…

## Stage 5 — Applications: Real-World SaaS Migrations

- Stripe Ledger Evolutions: Stripe uses a multi-phase column-mapping layer to move fields between database servers without dropping API requests.
- E-Commerce Account Merges: Moving customer shipping profiles from separate legacy databases into a unified schema during corporate mergers.
- High-Frequency Audits: Adding tracking metadata or audit hashes to core transaction records under live load.

*Applies **caching** to the decision above.*

**Observable outcome:** High-Frequency Audits: Adding tracking metadata or audit hashes to core transaction records under live load.

## Stage 6 — Practical Implementation: Liquibase & Spring Boot Code

- Let us implement the database migrations and application code for Phase 2 (Transition Phase) of our column rename operation.
- Database Migration Scripts (Liquibase XML) Step 1: Expand Phase (Add Column) Step 2: Contract Phase (Drop Column) 2\.
- Spring Boot JPA Transition Code During the transition phase, the application entity double-writes to both columns.
- We use JPA lifecycle callbacks ( @PrePersist and @PreUpdate ) to ensure that any change to the model updates both fields automatically.

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

```xml
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
 http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.3.xsd">

 <changeSet id="20260618-add-contact-phone-column" author="abstract-algorithms">
 <addColumn tableName="users">
 <column name="contact_phone" type="varchar(255)">
 <constraints nullable="true"/>
 </column>
 </addColumn>
 </changeSet>
</databaseChangeLog>
```

```xml
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
 http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.3.xsd">

 <changeSet id="20260618-drop-old-phone-number-column" author="abstract-algorithms">
 <dropColumn tableName="users" columnName="phone_number"/>
 </changeSet>
</databaseChangeLog>
```

**Observable outcome:** We use JPA lifecycle callbacks ( @PrePersist and @PreUpdate ) to ensure that any change to the model updates both fields automatically.

## End-to-end outcome

After working through the stages, you should be able to explain:

- Pattern Core: Expand (add new database field), Transition (application double-writes & copies history), Contract (drop old database field).
- Backfill Safety: Never run a single global UPDATE query on large tables.

## Alternatives considered

### Decision Guide: Expand-Contract vs. Locking Maintenance

Decision Guide: Expand-Contract vs. Locking Maintenance


## Failure and degraded modes

### Trade-offs and Failure Modes

- **Detection:** Operational Overhead: What would have been a single SQL script is split into multiple code deployments, data backfills, and database cleanup scripts.
- **User or operator impact:** Operational Overhead: What would have been a single SQL script is split into multiple code deployments, data backfills, and database cleanup scripts.

### Lessons Learned: Common Migration Pitfalls

- **Detection:** Running Backfill Before Deploying Double-Writes: If the backfill job runs while application instances are still writing only to the legacy column, any database records modified after they are backfil…
- **User or operator impact:** Running Backfill Before Deploying Double-Writes: If the backfill job runs while application instances are still writing only to the legacy column, any database records modified after they are backfil…

## Scenario context

Feature flags separate deploy from exposure. They are operationally valuable when you need cohort rollout, instant kill switches, or entitlement control without rebuilding or redeploying the service.

> **TLDR**: Flags help only when they are treated like production configuration with ownership, expiry, and observability. Otherwise they become a second codebase hidden behind conditionals.

> **Operator note:** Incid…

This walkthrough focuses on **When Feature Flags Actually Help**.

Before making a design choice, clarify:

- Feature flags are best when the deployment artifact and the exposure decision need to move at different speeds.
- Use them for: - controlled rollout by cohort, tenant, or region, - kill switches for risky integrations or expensive features, - entitlement and plan-based access control, - safe migration paths where new and old behavi…

**Concepts in play:** canary-deployments, jvm-internals-and-garbage-collection, caching, system-design-hld-architectures

## Stage 1 — How Flags Work in Production

- Good flag systems have two planes: 1.
- A control plane where owners define targeting rules, defaults, expiry, and audit history.
- A data plane where the application evaluates the flag locally or with a cached config snapshot.
- The production sequence usually looks like this: 1.

*Applies **canary-deployments** to the decision above.*

```mermaid
flowchart TD
    A[Draft - config only] --> B[Enabled - 1% rollout]
    B --> C[Ramp - 10% rollout]
    C --> D[Broad - 50% rollout]
    D --> E[Full - 100% rollout]
    E --> F[Archived - flag removed]
    B --> G[Disabled - rolled back]
    G --> B
    C --> G
    D --> G
```

**Observable outcome:** The production sequence usually looks like this: 1.

## Stage 2 — Unleash, LaunchDarkly OSS, and Flipt: Feature Flag Platforms in Practice

- Unleash is the leading open-source feature flag platform with a Java SDK, a rich strategy engine (gradual rollout, user targeting, custom constraints), A/B variant support, and a self-hostable control plane.
- Flipt is a lightweight, GitOps-friendly open-source flag server with a gRPC API.
- OpenFeature is a CNCF-incubated vendor-neutral SDK standard that decouples flag evaluation code from the backing provider.
- These tools solve the feature flag problem by providing a proper two-plane architecture: a control plane stores targeting rules, defaults, and audit history; a data plane evaluates flags locally from a cached snapshot s…

*Applies **jvm-internals-and-garbage-collection** to the decision above.*

```java
import io.getunleash.DefaultUnleash;
import io.getunleash.Unleash;
import io.getunleash.UnleashContext;
import io.getunleash.util.UnleashConfig;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class FeatureFlagConfig {

    @Bean
    public Unleash unleash() {
        // SDK polls the control plane every 15s and caches rules locally.
        // Evaluation never makes a live network call — the local cache answers.
        return new DefaultUnleash(
            UnleashConfig.builder()
                .appName("checkout-service")
                .instanceId(System.getenv().getOrDefault("HOSTNAME", "local"))
                .unleashAPI(System.getenv("UNLEASH_URL"))
                .apiKey(System.getenv("UNLEASH_TOKEN"))
                .build()
        );
    }
}

// Usage in any Spring bean — pass user/tenant context for targeting
boolean enabled = unleash.isEnabled(
    "new-checkout-flow",
    UnleashContext.builder()
        .userId(userId)
        .addProperty("plan", plan)
        .addProperty("region", region)
        .build(),
    false   // safe default if SDK cannot resolve the flag
);
```

**Observable outcome:** These tools solve the feature flag problem by providing a proper two-plane architecture: a control plane stores targeting rules, defaults, and audit history; a data plane evaluates flags locally from a cached snapshot s…

## Stage 3 — Enterprise Java Example: Rolling Out `checkout-risk-v2`

- Scenario: your checkout service has a new fraud/risk engine ( v2 ).
- You want to expose it only to enterprise tenants in eu-west at first, ramp gradually, and retain instant rollback.
- 1) Isolate the flag boundary in a dedicated component 2) Pass enterprise context into flag evaluation Control-plane targeting rule for this scenario: - Strategy 1: internal users = on - Strategy 2: plan=enterprise AND r…
- Every stale flag becomes hidden branch logic that on-call engineers must rediscover under pressure.

*Applies **caching** to the decision above.*

```java
package com.acme.checkout.flags;

import io.getunleash.DefaultUnleash;
import io.getunleash.Unleash;
import io.getunleash.util.UnleashConfig;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class FlagConfig {

  @Bean
  public Unleash unleash() {
    UnleashConfig config = UnleashConfig.builder()
        .appName("checkout-service")
        .instanceId(System.getenv().getOrDefault("HOSTNAME", "checkout-1"))
        .unleashAPI(System.getenv("UNLEASH_API_URL"))
        .apiKey(System.getenv("UNLEASH_API_TOKEN"))
        .build();

    return new DefaultUnleash(config);
  }
}
```

```java
package com.acme.checkout.flags;

import io.getunleash.Unleash;
import io.getunleash.UnleashContext;
import org.springframework.stereotype.Component;

@Component
public class FeatureDecisions {

  private final Unleash unleash;

  public FeatureDecisions(Unleash unleash) {
    this.unleash = unleash;
  }

  public boolean useRiskEngineV2(String userId, String tenantId, String plan, String region) {
    UnleashContext context = UnleashContext.builder()
        .userId(userId)
        .addProperty("tenant", tenantId)
        .addProperty("plan", plan)
        .addProperty("region", region)
        .build();

    // `false` is the safe default when flag state cannot be resolved.
    return unleash.isEnabled("checkout-risk-v2", context, false);
  }
}
```

**Observable outcome:** Every stale flag becomes hidden branch logic that on-call engineers must rediscover under pressure.

## End-to-end outcome

Feature flags are best when the deployment artifact and the exposure decision need to move at different speeds.

## Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

## Failure and degraded modes

### When Not to Use Feature Flags

- **Detection:** Flags are a poor substitute for basic code and architecture discipline.
- **User or operator impact:** Flags are a poor substitute for basic code and architecture discipline.



## Scenario context

Release safety is an architecture capability, not just a CI/CD convenience. Blue-green, canary, shadow traffic, feature flags, and GitOps patterns exist to control blast radius, measure regressions early, and make rollback fast enough to matter.

> **TLDR**: Safe deployments are controlled experiments: limit exposure, measure quickly, and make rollback boring.

### 🚨 The Problem This Solves

In 2021, a fintech rele…

This walkthrough focuses on **Why Deployment Patterns Belong in Architecture Reviews**.

Before making a design choice, clarify:

- Deployment design determines failure blast radius just as much as service design.
- If rollout controls are weak, good code still creates bad incidents.
- Practical review questions: - How fast can we detect regression?

**Concepts in play:** canary-deployments, caching, git-internals-and-versioning, api-gateway

## Stage 1 — When to Use Blue-Green, Canary, Shadow, Flags, and GitOps

- When not to overcomplicate - If service changes are low-risk and rare, basic canary may be enough.
- - If you cannot measure business impact, progressive rollout gives false confidence.
- 📊 Deployment Pipeline States This state diagram captures the distinct lifecycle states for two deployment strategies side by side.
- BlueGreen moves through GreenLive → BlueDeploy → BlueValidate → BlueLive in a hard cutover, while Canary progresses incrementally through small, broad, and full traffic slices before completing.

*Applies **canary-deployments** to the decision above.*

```mermaid
stateDiagram-v2
    state BlueGreen {
        [*] --> GreenLive
        GreenLive --> BlueDeploy
        BlueDeploy --> BlueValidate
        BlueValidate --> BlueLive
        BlueLive --> [*]
    }
    state Canary {
        [*] --> CanarySmall
        CanarySmall --> CanaryBroad
        CanaryBroad --> CanaryFull
        CanaryFull --> [*]
    }
```

**Observable outcome:** BlueGreen moves through GreenLive → BlueDeploy → BlueValidate → BlueLive in a hard cutover, while Canary progresses incrementally through small, broad, and full traffic slices before completing.

## Stage 2 — How the Release Control Loop Works

- Promote artifact to release candidate.
- Deploy through declarative desired state (GitOps or equivalent).
- Start canary slice and evaluate technical + business signals.
- Flip feature flags per cohort if needed.

*Applies **caching** to the decision above.*

**Observable outcome:** Flip feature flags per cohort if needed.

## Stage 3 — How to Implement: Progressive Delivery Checklist

- Define rollout gates (error, latency, saturation, business KPI).
- Define stop conditions and automatic rollback thresholds.
- Add traffic-routing primitives (weights or ring cohorts).
- Separate deploy from expose with feature flags.

*Applies **git-internals-and-versioning** to the decision above.*

**Observable outcome:** Separate deploy from expose with feature flags.

## Stage 4 — Deep Dive: Stateful Releases, Signal Quality, and Rollback Reality

- The Internals: Desired State + Runtime Gates GitOps controls desired state, but runtime safety still depends on gates and reversible data changes.
- Keep these concerns separate: - deployment : where code is running, - traffic : how much real traffic it receives, - feature exposure : which users see new behavior, - data compatibility : whether old and new versions c…
- Stateful change rule: never require immediate irreversible data transformation to keep serving.
- Performance Analysis: Metrics That Matter Most

*Applies **api-gateway** to the decision above.*

**Observable outcome:** Performance Analysis: Metrics That Matter Most

## Stage 5 — Rollout Flow: Deploy, Observe, Expand, or Revert

- This diagram shows the end-to-end progressive delivery loop from CI artifact through GitOps desired-state commit, controller deployment, shadow checks, and staged canary traffic expansion.
- The gate check after initial canary exposure is the critical decision point: a passing gate expands ring by ring until feature flags complete exposure, while a failing gate triggers an immediate rollback of both traffic…
- The key takeaway is that safety comes from gating every expansion step, not from deploying slowly.
- 📊 Traffic Routing Comparison This diagram contrasts how traffic is split across BlueGreen, Canary, and Shadow strategies at the load balancer level.

*Applies **canary-deployments** to the decision above.*

```mermaid
flowchart TD
  A[CI artifact] --> B[GitOps desired state commit]
  B --> C[Controller deploys candidate]
  C --> D[Shadow checks and smoke tests]
  D --> E[Canary 1-5 percent traffic]
  E --> F{Gates pass?}
  F -->|Yes| G[Expand traffic ring by ring]
  G --> H[Enable feature flags by cohort]
  F -->|No| I[Rollback traffic and release]
```

```mermaid
flowchart LR
    subgraph BlueGreen
        LB1[Load Balancer] -->|100%| BG1[Blue v1]
        LB1 -.->|0% cutover| BG2[Green v2]
    end
    subgraph Canary
        LB2[Load Balancer] -->|95%| C1[Stable v1]
        LB2 -->|5%| C2[Canary v2]
    end
    subgraph Shadow
        LB3[Load Balancer] -->|100%| S1[Live v1]
        LB3 -.->|mirror| S2[Shadow v2]
    end
```

**Observable outcome:** 📊 Traffic Routing Comparison This diagram contrasts how traffic is split across BlueGreen, Canary, and Shadow strategies at the load balancer level.

## Stage 6 — Practical Example: Canary Policy With Automatic Abort

- The safest rollout controllers encode traffic steps and abort conditions directly in config so the happy path and the rollback path use the same source of truth.
- Operational checks that matter more than the syntax: 1.
- The pause window has to be longer than the metric stabilization window, or the gate is decorative.
- Technical and business guardrails should both participate in abort decisions.

*Applies **caching** to the decision above.*

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: recommendation-api
spec:
  replicas: 12
  strategy:
    canary:
      maxUnavailable: 0
      canaryService: recommendation-api-canary
      stableService: recommendation-api-stable
      steps:
        - setWeight: 5
        - paus
e:
            duration: 10m
        - analysi
s:
            templates:
              - templateName: canary-errors
              - templateName: conversion-guardrail
        - setWeight: 25
        - paus
e:
            duration: 20m
```

**Observable outcome:** Technical and business guardrails should both participate in abort decisions.

## End-to-end outcome

After working through the stages, you should be able to explain:

- - Deploy and expose are different control planes and should stay separate.
- - Canary and shadow only work with representative traffic and meaningful gates.

## Alternatives considered

### Decision Guide: Picking a Rollout Pattern Fast

Use combinations deliberately, not by default.


## Failure and degraded modes

### Trade-offs & Failure Modes: Pros, Cons, and Risks

- **Detection:** Trade-offs & Failure Modes: Pros, Cons, and Risks
- **User or operator impact:** Trade-offs & Failure Modes: Pros, Cons, and Risks



Evolve legacy systems safely with rollout and migration patterns.

Modernization and Delivery

A practical guide to enterprise backend patterns: DDD, BFF, CQRS, event sourcing, sagas, outbox, feature flags, and deployment workflows.