How CDC Works Across Databases: PostgreSQL, MySQL, MongoDB, and Beyond
Where CDC Doesn't Work: Warehouses, Serverless, and Blocked Managed Databases
A data engineering team at a fintech company built what they believed was a robust Change Data Capture pipeline: three source databases (PostgreSQL, MongoDB, and Cassandra), Debezium connectors wired to Kafka, and a downstream data warehouse receiving every insert, update, and delete.
🎯 System Scenario & Requirements
System Goal
A data engineering team at a fintech company built what they believed was a robust Change Data Capture pipeline: three source databases (PostgreSQL, MongoDB, and Cassandra), Debezium connectors wired to Kafka, and a downstream data warehouse receiving every insert, update, and delete.
Actors & Components
- Application Workers
- Volatile Memory Tier
- Persistent Disk Tier
- Worker Process
- In-Memory Cache
- Primary Database
Functional Requirements
- A data engineering team at a fintech company built what they believed was a robust Change Data Capture pipeline: three source databases (PostgreSQL, MongoDB, and Cassandra), Debezium connectors wired to Kafka, and a dow…
- Six months into production they discovered three silent failures: their Cassandra CDC had been silently converting deletes into nothing — because Cassandra's commitlog CDC does not emit delete notifications in the way a…
- Their MongoDB connector had been failing to capture changes on one of the clusters — because that cluster was a standalone instance, not a replica set, and MongoDB CDC simply does not work on standalone deployments.
Constraints & Assumptions
- The diagram below maps the log retention lifecycle for three representative databases side by side, showing where each mechanism's failure boundary lies.
- Understanding these boundaries is the first step to building operational alerting that catches problems before they become unrecoverable.
- This example revisits the fintech team from the opening — their PostgreSQL, MongoDB, and Cassandra pipeline — and traces exactly where each failure originated and how it was diagnosed.
- The team's pipeline processed account ledger entries (PostgreSQL), user activity documents (MongoDB), and session key-value data (Cassandra).
⚡ Execution Stages & Decision Trace
Stage 1 — Where These CDC Patterns Power Production Systems at Scale
Understanding the mechanisms abstractly is useful, but seeing where these patterns appear in real production systems reveals why the configuration details matter.
LinkedIn's Databus (origin of modern CDC) was built to capture changes from Oracle databases and feed them to search indexes, social graph caches, and recommendations systems.
Deep Dive
Scenario context
A data engineering team at a fintech company built what they believed was a robust Change Data Capture pipeline: three source databases (PostgreSQL, MongoDB, and Cassandra), Debezium connectors wired to Kafka, and a downstream data warehouse receiving every insert, update, and delete. Six months into production they discovered three silent failures: their Cassandra CDC had been silently converting deletes into nothi…
This walkthrough focuses on Where CDC Doesn't Work: Warehouses, Serverless, and Blocked Managed Databases.
Before making a design choice, clarify:
- Not every database exposes a log that external readers can access.
- The following categories either lack a log-based CDC mechanism entirely or expose it only within the database's own ecosystem.
- Google BigQuery is an analytical data warehouse with no transaction log exposed to users.
Concepts in play: event-driven-architecture, consistency-replication, btree-vs-lsmtree, caching
Stage 1 — Where These CDC Patterns Power Production Systems at Scale
- Understanding the mechanisms abstractly is useful, but seeing where these patterns appear in real production systems reveals why the configuration details matter.
- LinkedIn's Databus (origin of modern CDC) was built to capture changes from Oracle databases and feed them to search indexes, social graph caches, and recommendations systems.
- Databus exposed the problem of Oracle LogMiner latency at scale: the redo log parsing latency meant that profile updates could be invisible to search for 15–30 seconds.
- LinkedIn ultimately moved portions of their stack to MySQL binlog-based CDC to reduce latency.
Observable outcome: LinkedIn ultimately moved portions of their stack to MySQL binlog-based CDC to reduce latency.
Stage 2 — Debezium: The Open-Source Engine That Reads Database Logs
- Debezium is the primary open-source CDC framework used in production across all the database types covered in this post.
- It implements log-based CDC connectors for PostgreSQL (logical decoding), MySQL (binlog streaming), MongoDB (change streams), Oracle (LogMiner), SQL Server (built-in CDC API), and Cassandra (commitlog), among others.
- Debezium connectors run as Kafka Connect source connectors and publish change events to Kafka topics.
- Each Debezium connector handles the database-specific protocol differences described in this post: creating and managing replication slots for PostgreSQL, managing binlog position and GTID sets for MySQL, managing chang…
Observable outcome: Each Debezium connector handles the database-specific protocol differences described in this post: creating and managing replication slots for PostgreSQL, managing binlog position and GTID sets for MySQL, managing chang…
End-to-end outcome
After working through the stages, you should be able to explain:
- Lesson 1: Monitor log retention lag, not just connector status.
- Every database-level CDC mechanism has a retention boundary.
Alternatives considered
Database CDC Capability Comparison
The table below summarizes the CDC mechanism, key requirements, and critical operational gotchas for each database covered in this post.
Choosing Your CDC Approach: A Decision Guide by Database and Constraint
The right CDC approach is determined by four constraints: the source database engine, whether you are running on a managed cloud service (and which tier), how much lag is acceptable, and whether you need full delete semantics.
Failure and degraded modes
From Write to Consumer: Visualizing the CDC Failure Boundaries
- Detection: The diagram below maps the log retention lifecycle for three representative databases side by side, showing where each mechanism's failure boundary lies.
- User or operator impact: The diagram below maps the log retention lifecycle for three representative databases side by side, showing where each mechanism's failure boundary lies.
Worked Example: Three Databases, Three Silent Failures, One Pipeline
- Detection: This example revisits the fintech team from the opening — their PostgreSQL, MongoDB, and Cassandra pipeline — and traces exactly where each failure originated and how it was diagnosed.
- User or operator impact: This example revisits the fintech team from the opening — their PostgreSQL, MongoDB, and Cassandra pipeline — and traces exactly where each failure originated and how it was diagnosed.