Big Data Architecture Patterns: Lambda, Kappa, CDC, Medallion, and Data Mesh
Scenario context
A serious data platform is defined less by where files are stored and more by how changes enter the system, how serving layers are materialized, and who owns quality over time.
🎯 System Scenario & Requirements
System Goal
A serious data platform is defined less by where files are stored and more by how changes enter the system, how serving layers are materialized, and who owns quality over time.
Actors & Components
- Application or API tier
- Core service under design
- Operator or SRE observer
Functional Requirements
- A serious data platform is defined less by where files are stored and more by how changes enter the system, how serving layers are materialized, and who owns quality over time.
- Lambda, Kappa, CDC, Medallion, and Data Mesh are patterns for making those choices explicit.
- How it solves the problem: The PySpark Medallion snippet in the Practical Example section above uses Delta Lake's MERGE INTO for idempote… This walkthrough focuses on Lessons Learned .
Constraints & Assumptions
- The source does not isolate a separate failure mode.
- Limits that matter for the scenario are called out in the stage decisions above.
⚡ Execution Stages & Decision Trace
Stage 1 — End-to-end outcome
After working through the stages, you should be able to explain: - - Pattern selection should be driven by freshness, replay, and ownership requirements.
- - CDC and Medallion are often the most practical first moves.
Deep Dive
Scenario context
A serious data platform is defined less by where files are stored and more by how changes enter the system, how serving layers are materialized, and who owns quality over time. Lambda, Kappa, CDC, Medallion, and Data Mesh are patterns for making those choices explicit.
How it solves the problem: The PySpark Medallion snippet in the Practical Example section above uses Delta Lake's MERGE INTO for idempote…
This walkthrough focuses on Scenario context.
Before making a design choice, clarify:
- A serious data platform is defined less by where files are stored and more by how changes enter the system, how serving layers are materialized, and who owns quality over time.
- Lambda, Kappa, CDC, Medallion, and Data Mesh are patterns for making those choices explicit.
- How it solves the problem: The PySpark Medallion snippet in the Practical Example section above uses Delta Lake's MERGE INTO for idempote… This walkthrough focuses on Lessons Learned .
Concepts in play: data-warehousing-lakehouse, event-driven-architecture, spark-data-engineering
Stage 1 — End-to-end outcome
- After working through the stages, you should be able to explain: - - Pattern selection should be driven by freshness, replay, and ownership requirements.
- CDC and Medallion are often the most practical first moves.
Observable outcome: - - CDC and Medallion are often the most practical first moves.
End-to-end outcome
A serious data platform is defined less by where files are stored and more by how changes enter the system, how serving layers are materialized, and who owns quality over time.
Alternatives considered
Alternatives considered
The source focuses on one primary design.
Failure and degraded modes
Failure and degraded modes
- Detection: The source does not isolate a separate failure mode.
- User or operator impact: The source does not isolate a separate failure mode.
Key learnings
- After working through the stages, you should be able to explain: - - Pattern selection should be driven by freshness, replay, and ownership requirements.