Skip to main content
← PROBLEMS / SYSTEM-DESIGN
An end-to-end domain learning journey

Design an Idempotent Financial Settlement
DAG Pipeline with Atomic WAP.

Partition settlement runs by (settlement_date, run_id) into Iceberg WAP staging branches, enforce conservation-of-money audits, and atomically fast-forward to production.
Start the journey7 stages
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Design a daily and hourly financial settlement DAG processing 500M ledger events/day across 200 Spark executors with zero partial-success visibility, deterministic idempotency on Airflow retries, and automated quarantine on balance-conservation failures.

From patterns to decisions.

Partition settlement runs by `(settlement_date, run_id)` into Iceberg WAP staging branches, enforce conservation-of-money audits, and atomically fast-forward to production.

Full walkthrough · 7 stages
01
STAGE 01 · CORE INTUITIONfoundation

Scale Envelope & Latency Budget.

Design a daily and hourly financial settlement DAG processing 500M ledger events/day across 200 Spark executors with zero partial-success visibility, deterministic idempotency on Airflow retries, and automated quarantine on balance-conservation failures.

STAGE 01 · CORE INTUITION

Scale Envelope & Latency Budget

Scale EnvelopeIngress / BaselineLatency BudgetScale ConstraintValidation GateBottleneck SignalProduction OutcomeTarget SLAexposes

1.Stage Objective: Size throughput, storage, and p99 latency targets.

2.Level (Foundation): Design a daily and hourly financial settlement DAG processing 500M ledger events/day across 200 Spark executors with zero partial-success vi

3.Operational Invariant: Size throughput, storage, and p99 latency targets.

Caption: Size throughput, storage, and p99 latency targets.

02
STAGE 02 · CORE MECHANICSfoundation

Core Architecture & Data Flow.

Using Apache Iceberg Write-Audit-Publish (WAP), each run_id writes deterministically to a staging branch, runs automated Great Expectations / SQL invariant checks (SUM(debit) == SUM(credit), zero duplicate tx_id), and atomically replaces the target partition on main.

STAGE 02 · CORE MECHANICS

Core Architecture & Data Flow

Core ArchitectureInput StateData FlowCore MechanismValidation GateInvariant CheckProduction OutcomeOutput Statetransforms

1.Stage Objective: Assemble the primary data path and storage layout.

2.Level (Foundation): Using Apache Iceberg Write-Audit-Publish (WAP), each `run_id` writes deterministically to a staging branch, runs automated Great Expectation

3.Operational Invariant: Assemble the primary data path and storage layout.

Caption: Assemble the primary data path and storage layout.

03
STAGE 03 · SAFE IMPLEMENTATIONintermediate

Idempotent Partition Overwrite & Audit Gate.

def settle_partition_idempotent(spark, catalog, partition_date: str, run_id: str) -> None:
    branch = f'settle_{partition_date}_{run_id}'
    catalog.create_or_reset_branch(branch)
    spark.compute_settlement(partition_date).write_iceberg(branch=branch, mode='overwrite_partition')
    audit = spark.audit_conservation(branch, partition_date)
    if audit['imbalance_cents'] != 0 or audit['dup_tx_count'] > 0:
        catalog.quarantine(branch)
        raise ValueError(f'Settlement audit failed: {audit}')
    catalog.atomic_publish(branch, target='main')
STAGE 03 · SAFE IMPLEMENTATION

Idempotent Partition Overwrite & Audit Gate

IdempotentPartition OverwriValidated InputAudit GateAtomic StepValidation GateError / Retry GuardProduction OutcomeCommitted Resultexecutes

1.Stage Objective: Guarantee retry safety via deterministic staging branch cleanup and atomic partition replace.

2.Level (Intermediate): def settle_partition_idempotent(spark, catalog, partition_date: str, run_id: str) -> None:

3.Operational Invariant: Guarantee retry safety via deterministic staging branch cleanup and atomic partition replace.

Caption: Guarantee retry safety via deterministic staging branch cleanup and atomic partition replace.

04
STAGE 04 · SCALING & TRADE-OFFSintermediate

Sharding & Hot-Spot Mitigation.

Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.

STAGE 04 · SCALING & TRADE-OFFS

Sharding & Hot-Spot Mitigation

ShardingHot PathHot-Spot MitigationPartition / IndexValidation GateLatency / RAM BoundProduction OutcomeScaled Statebounds

1.Stage Objective: Prevent single-partition bottlenecks at peak load.

2.Level (Intermediate): Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.

3.Operational Invariant: Prevent single-partition bottlenecks at peak load.

Caption: Prevent single-partition bottlenecks at peak load.

05
STAGE 05 · FAULT ISOLATIONadvanced

Failure Modes & Graceful Degradation.

Apply circuit breakers, fallback caches, and dead-letter quarantine queues.

STAGE 05 · FAULT ISOLATION

Failure Modes & Graceful Degradation

Failure ModesFault TriggerGraceful DegradationIsolation BoundaryValidation GateFallback / QuarantineProduction OutcomeProtected Pathisolates

1.Stage Objective: Keep critical paths healthy during partial outages.

2.Level (Advanced): Apply circuit breakers, fallback caches, and dead-letter quarantine queues.

3.Operational Invariant: Keep critical paths healthy during partial outages.

Caption: Keep critical paths healthy during partial outages.

06
STAGE 06 · RECOVERY & CONSISTENCYadvanced

Consistency, Idempotency & Recovery.

Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.

STAGE 06 · RECOVERY & CONSISTENCY

Consistency, Idempotency & Recovery

ConsistencyIdempotencyCheckpoint / LogRecoveryReplay / SyncValidation GateConsistency GateProduction OutcomeRecovered Stateverifies

1.Stage Objective: Guarantee correctness across retries and node crashes.

2.Level (Advanced): Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.

3.Operational Invariant: Guarantee correctness across retries and node crashes.

Caption: Guarantee correctness across retries and node crashes.

07
STAGE 07 · OBSERVABILITY & GUARDRAILSadvanced

Telemetry, Circuit Breakers & Failure Modes.

Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants degrade.

STAGE 07 · OBSERVABILITY & GUARDRAILS

Production Guardrails

ProductionGuardrailsTelemetry SignalCore EngineSLO ThresholdGuard & VerifyCircuit BreakerProduction OutcomeSafe Degradationtrips

1.Stage Objective: Monitor tail latency, error budgets, and invariant violations in production.

2.Level (Advanced): Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants de

3.Operational Invariant: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.

Caption: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Question 1 of 3
Score: 0/3

How do you ensure Airflow task retries do not duplicate settlement rows?

Recommended Learning Flow