Skip to main content
← PROBLEMS / CONCURRENCY
An end-to-end domain learning journey

Design an Atomic Payment Ledger &
Reservation Engine Preventing TOCTOU Races.

Combine an atomic INSERT ... ON CONFLICT idempotency gate with single-statement conditional UPDATE + version CAS and an immutable double-entry ledger table.
Start the journey7 stages
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Design a payment reservation and double-entry ledger service handling 25K concurrent debits/sec across 10M accounts with zero negative-balance TOCTOU races, strict idempotency on client retries, and full auditability.

From patterns to decisions.

Combine an atomic `INSERT ... ON CONFLICT` idempotency gate with single-statement conditional `UPDATE` + version CAS and an immutable double-entry ledger table.

Full walkthrough · 7 stages
01
STAGE 01 · CORE INTUITIONfoundation

Scale Envelope & Latency Budget.

Design a payment reservation and double-entry ledger service handling 25K concurrent debits/sec across 10M accounts with zero negative-balance TOCTOU races, strict idempotency on client retries, and full auditability.

STAGE 01 · CORE INTUITION

Scale Envelope & Latency Budget

Scale EnvelopeIngress / BaselineLatency BudgetScale ConstraintValidation GateBottleneck SignalProduction OutcomeTarget SLAexposes

1.Stage Objective: Size throughput, storage, and p99 latency targets.

2.Level (Foundation): Design a payment reservation and double-entry ledger service handling 25K concurrent debits/sec across 10M accounts with zero negative-balan

3.Operational Invariant: Size throughput, storage, and p99 latency targets.

Caption: Size throughput, storage, and p99 latency targets.

02
STAGE 02 · CORE MECHANICSfoundation

Core Architecture & Data Flow.

The hardened architecture enforces three atomic layers: (1) an Idempotency-Key table claimed via INSERT ... ON CONFLICT DO NOTHING + SHA-256 payload hash check; (2) a single conditional UPDATE ledger_accounts SET balance = balance - :amt, version = version + 1 WHERE id = :id AND balance >= :amt AND version = :v; and (3) an append-only ledger_entries insert in the same local PostgreSQL transaction.

STAGE 02 · CORE MECHANICS

Core Architecture & Data Flow

Core ArchitectureInput StateData FlowCore MechanismValidation GateInvariant CheckProduction OutcomeOutput Statetransforms

1.Stage Objective: Assemble the primary data path and storage layout.

2.Level (Foundation): The hardened architecture enforces three atomic layers: (1) an `Idempotency-Key` table claimed via `INSERT ... ON CONFLICT DO NOTHING` + SHA

3.Operational Invariant: Assemble the primary data path and storage layout.

Caption: Assemble the primary data path and storage layout.

03
STAGE 03 · SAFE IMPLEMENTATIONintermediate

Atomic Conditional Debit & Idempotency Gate.

def execute_atomic_debit(cur, account_id: str, amount: int, idem_key: str, payload_hash: str) -> dict:
    if amount <= 0:
        raise ValueError('Debit amount must be positive')
    cur.execute(
        'INSERT INTO idempotency_keys(key, payload_hash, status) VALUES (%s, %s, %s) ON CONFLICT (key) DO NOTHING',
        (idem_key, payload_hash, 'IN_PROGRESS'),
    )
    if cur.rowcount == 0:
        return {'status': 'DUPLICATE_OR_IN_FLIGHT'}
    cur.execute(
        'UPDATE accounts SET balance = balance - %s, version = version + 1 WHERE id = %s AND balance >= %s RETURNING balance, version',
        (amount, account_id, amount),
    )
    row = cur.fetchone()
    if row is None:
        raise ValueError('Insufficient funds or missing account')
    return {'status': 'COMMITTED', 'balance': row[0], 'version': row[1]}
STAGE 03 · SAFE IMPLEMENTATION

Atomic Conditional Debit & Idempotency Gate

Atomic ConditionalDebitValidated InputIdempotency GateAtomic StepValidation GateError / Retry GuardProduction OutcomeCommitted Resultexecutes

1.Stage Objective: Claim idempotency and execute conditional balance debit in one ACID transaction.

2.Level (Intermediate): def execute_atomic_debit(cur, account_id: str, amount: int, idem_key: str, payload_hash: str) -> dict:

3.Operational Invariant: Claim idempotency and execute conditional balance debit in one ACID transaction.

Caption: Claim idempotency and execute conditional balance debit in one ACID transaction.

04
STAGE 04 · SCALING & TRADE-OFFSintermediate

Sharding & Hot-Spot Mitigation.

Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.

STAGE 04 · SCALING & TRADE-OFFS

Sharding & Hot-Spot Mitigation

ShardingHot PathHot-Spot MitigationPartition / IndexValidation GateLatency / RAM BoundProduction OutcomeScaled Statebounds

1.Stage Objective: Prevent single-partition bottlenecks at peak load.

2.Level (Intermediate): Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.

3.Operational Invariant: Prevent single-partition bottlenecks at peak load.

Caption: Prevent single-partition bottlenecks at peak load.

05
STAGE 05 · FAULT ISOLATIONadvanced

Failure Modes & Graceful Degradation.

Apply circuit breakers, fallback caches, and dead-letter quarantine queues.

STAGE 05 · FAULT ISOLATION

Failure Modes & Graceful Degradation

Failure ModesFault TriggerGraceful DegradationIsolation BoundaryValidation GateFallback / QuarantineProduction OutcomeProtected Pathisolates

1.Stage Objective: Keep critical paths healthy during partial outages.

2.Level (Advanced): Apply circuit breakers, fallback caches, and dead-letter quarantine queues.

3.Operational Invariant: Keep critical paths healthy during partial outages.

Caption: Keep critical paths healthy during partial outages.

06
STAGE 06 · RECOVERY & CONSISTENCYadvanced

Consistency, Idempotency & Recovery.

Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.

STAGE 06 · RECOVERY & CONSISTENCY

Consistency, Idempotency & Recovery

ConsistencyIdempotencyCheckpoint / LogRecoveryReplay / SyncValidation GateConsistency GateProduction OutcomeRecovered Stateverifies

1.Stage Objective: Guarantee correctness across retries and node crashes.

2.Level (Advanced): Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.

3.Operational Invariant: Guarantee correctness across retries and node crashes.

Caption: Guarantee correctness across retries and node crashes.

07
STAGE 07 · OBSERVABILITY & GUARDRAILSadvanced

Telemetry, Circuit Breakers & Failure Modes.

Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants degrade.

STAGE 07 · OBSERVABILITY & GUARDRAILS

Production Guardrails

ProductionGuardrailsTelemetry SignalCore EngineSLO ThresholdGuard & VerifyCircuit BreakerProduction OutcomeSafe Degradationtrips

1.Stage Objective: Monitor tail latency, error budgets, and invariant violations in production.

2.Level (Advanced): Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants de

3.Operational Invariant: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.

Caption: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Question 1 of 3
Score: 0/3

Why is `UPDATE accounts SET balance = balance - amt WHERE id = ? AND balance >= amt` immune to TOCTOU?

Recommended Learning Flow