Change Feed vs Change Stream: CDC Internals, Reliability, and When to Avoid Each
Ordering Across Partitions: Why Global Order Is a Coordination Problem
In the summer of 2023, the platform team at a fast-growing e-commerce company was handling 100,000 orders per day across three microservices: Order Service, Inventory Service, and Billing Service.
🎯 System Scenario & Requirements
System Goal
In the summer of 2023, the platform team at a fast-growing e-commerce company was handling 100,000 orders per day across three microservices: Order Service, Inventory Service, and Billing Service.
Actors & Components
- Application or API tier
- Core service under design
- Operator or SRE observer
Functional Requirements
- In the summer of 2023, the platform team at a fast-growing e-commerce company was handling 100,000 orders per day across three microservices: Order Service, Inventory Service, and Billing Service.
- All three needed to react to the same database mutations — when an order was placed, inventory had to be reserved; when it was confirmed, billing had to be triggered; when it was cancelled, inventory had to be released.
- The solution they shipped was polling: each downstream service ran a background thread that executed SELECT FROM orders WHERE updated at :last check every five seconds.
Constraints & Assumptions
- Six failure patterns account for the majority of production CDC incidents.
- Each has a specific detection method and a specific recovery path.
⚡ Execution Stages & Decision Trace
Stage 1 — When Change Feeds and Streams Are the Right Architectural Tool
CDC is the correct choice for these patterns, and applying it correctly here saves engineering effort that polling, dual-writes, or synchronous hooks cannot avoid.
Event-driven microservice choreography : An order is inserted → the Inventory Service reserves stock → the Billing Service creates an invoice → the Notification Service sends a confirmation email.
Deep Dive
Scenario context
In the summer of 2023, the platform team at a fast-growing e-commerce company was handling 100,000 orders per day across three microservices: Order Service, Inventory Service, and Billing Service. All three needed to react to the same database mutations — when an order was placed, inventory had to be reserved; when it was confirmed, billing had to be triggered; when it was cancelled, inventory had to be released. Th…
This walkthrough focuses on Ordering Across Partitions: Why Global Order Is a Coordination Problem.
Before making a design choice, clarify:
- A common and costly misconception is that CDC gives you a globally ordered stream of all changes across all tables and all partitions.
- With one partial exception, it does not.
- Understanding why helps you design consumers that degrade gracefully when events arrive out of order.
Concepts in play: event-driven-architecture, jvm-internals-and-garbage-collection, database-sharding, consistency-replication
Stage 1 — When Change Feeds and Streams Are the Right Architectural Tool
- CDC is the correct choice for these patterns, and applying it correctly here saves engineering effort that polling, dual-writes, or synchronous hooks cannot avoid.
- Event-driven microservice choreography : An order is inserted → the Inventory Service reserves stock → the Billing Service creates an invoice → the Notification Service sends a confirmation email.
- Each service subscribes to the orders change feed and reacts to the relevant op: 'c' (insert) or op: 'u' (update) events.
- No synchronous service-to-service HTTP calls, no shared transaction boundaries, no tight coupling.
Observable outcome: No synchronous service-to-service HTTP calls, no shared transaction boundaries, no tight coupling.
Stage 2 — When to Avoid Change Feeds and Streams
- CDC is not always the right tool.
- Using it in these scenarios introduces unnecessary operational complexity or actively makes the problem worse.
- High-frequency ephemeral state updates : A multiplayer game writing 10,000 position updates per second per player does not need every position update propagated to downstream services — it needs the latest position.
- Emitting every game state mutation through a CDC pipeline produces massive Kafka topic lag, stale data by the time consumers process events, and very high storage costs for events that have no durable business significa…
Observable outcome: Emitting every game state mutation through a CDC pipeline produces massive Kafka topic lag, stale data by the time consumers process events, and very high storage costs for events that have no durable business significa…
Stage 3 — Full System Architecture: From Database Commit to Downstream Consumer with Recovery Paths
- The following diagram shows a production CDC architecture end-to-end: source database through a CDC connector to a Kafka topic, fanned out to three downstream consumers, with failure recovery paths and dead-letter handl…
- This is the architecture that replaces the broken polling approach from the opening scenario.
- Retention windows are the most common source of silent data loss.
- DynamoDB's 24-hour window has surprised more than one team whose consumer went offline over a long weekend.
flowchart TD
DB["Database
(WAL/Log)"]
Capture["Capture Engine"]
Transform["Transform"]
Sink["Sink/Destination"]
DB -->|"Change Log"| Capture
Capture -->|"CDC Event"| Transform
Transform -->|"Normalized"| Sink
Observable outcome: DynamoDB's 24-hour window has surprised more than one team whose consumer went offline over a long weekend.
End-to-end outcome
A common and costly misconception is that CDC gives you a globally ordered stream of all changes across all tables and all partitions.
Alternatives considered
The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.
Failure and degraded modes
Common Failure Modes and How to Recover Before They Escalate
- Detection: Six failure patterns account for the majority of production CDC incidents.
- User or operator impact: Six failure patterns account for the majority of production CDC incidents.