Design a daily and hourly financial settlement DAG processing 500M ledger events/day across 200 Spark executors with zero partial-success visibility, deterministic idempotency on Airflow retries, and automated quarantine on balance-conservation failures.
From patterns to decisions.
Partition settlement runs by `(settlement_date, run_id)` into Iceberg WAP staging branches, enforce conservation-of-money audits, and atomically fast-forward to production.
Scale Envelope & Latency Budget.
Design a daily and hourly financial settlement DAG processing 500M ledger events/day across 200 Spark executors with zero partial-success visibility, deterministic idempotency on Airflow retries, and automated quarantine on balance-conservation failures.
Scale Envelope & Latency Budget
1.Stage Objective: Size throughput, storage, and p99 latency targets.
2.Level (Foundation): Design a daily and hourly financial settlement DAG processing 500M ledger events/day across 200 Spark executors with zero partial-success vi
3.Operational Invariant: Size throughput, storage, and p99 latency targets.
Caption: Size throughput, storage, and p99 latency targets.
Core Architecture & Data Flow.
Using Apache Iceberg Write-Audit-Publish (WAP), each run_id writes deterministically to a staging branch, runs automated Great Expectations / SQL invariant checks (SUM(debit) == SUM(credit), zero duplicate tx_id), and atomically replaces the target partition on main.
Core Architecture & Data Flow
1.Stage Objective: Assemble the primary data path and storage layout.
2.Level (Foundation): Using Apache Iceberg Write-Audit-Publish (WAP), each `run_id` writes deterministically to a staging branch, runs automated Great Expectation
3.Operational Invariant: Assemble the primary data path and storage layout.
Caption: Assemble the primary data path and storage layout.
Idempotent Partition Overwrite & Audit Gate.
def settle_partition_idempotent(spark, catalog, partition_date: str, run_id: str) -> None:
branch = f'settle_{partition_date}_{run_id}'
catalog.create_or_reset_branch(branch)
spark.compute_settlement(partition_date).write_iceberg(branch=branch, mode='overwrite_partition')
audit = spark.audit_conservation(branch, partition_date)
if audit['imbalance_cents'] != 0 or audit['dup_tx_count'] > 0:
catalog.quarantine(branch)
raise ValueError(f'Settlement audit failed: {audit}')
catalog.atomic_publish(branch, target='main')Idempotent Partition Overwrite & Audit Gate
1.Stage Objective: Guarantee retry safety via deterministic staging branch cleanup and atomic partition replace.
2.Level (Intermediate): def settle_partition_idempotent(spark, catalog, partition_date: str, run_id: str) -> None:
3.Operational Invariant: Guarantee retry safety via deterministic staging branch cleanup and atomic partition replace.
Caption: Guarantee retry safety via deterministic staging branch cleanup and atomic partition replace.
Sharding & Hot-Spot Mitigation.
Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.
Sharding & Hot-Spot Mitigation
1.Stage Objective: Prevent single-partition bottlenecks at peak load.
2.Level (Intermediate): Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.
3.Operational Invariant: Prevent single-partition bottlenecks at peak load.
Caption: Prevent single-partition bottlenecks at peak load.
Failure Modes & Graceful Degradation.
Apply circuit breakers, fallback caches, and dead-letter quarantine queues.
Failure Modes & Graceful Degradation
1.Stage Objective: Keep critical paths healthy during partial outages.
2.Level (Advanced): Apply circuit breakers, fallback caches, and dead-letter quarantine queues.
3.Operational Invariant: Keep critical paths healthy during partial outages.
Caption: Keep critical paths healthy during partial outages.
Consistency, Idempotency & Recovery.
Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.
Consistency, Idempotency & Recovery
1.Stage Objective: Guarantee correctness across retries and node crashes.
2.Level (Advanced): Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.
3.Operational Invariant: Guarantee correctness across retries and node crashes.
Caption: Guarantee correctness across retries and node crashes.
Telemetry, Circuit Breakers & Failure Modes.
Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants degrade.
Production Guardrails
1.Stage Objective: Monitor tail latency, error budgets, and invariant violations in production.
2.Level (Advanced): Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants de
3.Operational Invariant: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.
Caption: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.
Practice: Make the trade-off yourself.
How do you ensure Airflow task retries do not duplicate settlement rows?