Design a payment reservation and double-entry ledger service handling 25K concurrent debits/sec across 10M accounts with zero negative-balance TOCTOU races, strict idempotency on client retries, and full auditability.
From patterns to decisions.
Combine an atomic `INSERT ... ON CONFLICT` idempotency gate with single-statement conditional `UPDATE` + version CAS and an immutable double-entry ledger table.
Scale Envelope & Latency Budget.
Design a payment reservation and double-entry ledger service handling 25K concurrent debits/sec across 10M accounts with zero negative-balance TOCTOU races, strict idempotency on client retries, and full auditability.
Scale Envelope & Latency Budget
1.Stage Objective: Size throughput, storage, and p99 latency targets.
2.Level (Foundation): Design a payment reservation and double-entry ledger service handling 25K concurrent debits/sec across 10M accounts with zero negative-balan
3.Operational Invariant: Size throughput, storage, and p99 latency targets.
Caption: Size throughput, storage, and p99 latency targets.
Core Architecture & Data Flow.
The hardened architecture enforces three atomic layers: (1) an Idempotency-Key table claimed via INSERT ... ON CONFLICT DO NOTHING + SHA-256 payload hash check; (2) a single conditional UPDATE ledger_accounts SET balance = balance - :amt, version = version + 1 WHERE id = :id AND balance >= :amt AND version = :v; and (3) an append-only ledger_entries insert in the same local PostgreSQL transaction.
Core Architecture & Data Flow
1.Stage Objective: Assemble the primary data path and storage layout.
2.Level (Foundation): The hardened architecture enforces three atomic layers: (1) an `Idempotency-Key` table claimed via `INSERT ... ON CONFLICT DO NOTHING` + SHA
3.Operational Invariant: Assemble the primary data path and storage layout.
Caption: Assemble the primary data path and storage layout.
Atomic Conditional Debit & Idempotency Gate.
def execute_atomic_debit(cur, account_id: str, amount: int, idem_key: str, payload_hash: str) -> dict:
if amount <= 0:
raise ValueError('Debit amount must be positive')
cur.execute(
'INSERT INTO idempotency_keys(key, payload_hash, status) VALUES (%s, %s, %s) ON CONFLICT (key) DO NOTHING',
(idem_key, payload_hash, 'IN_PROGRESS'),
)
if cur.rowcount == 0:
return {'status': 'DUPLICATE_OR_IN_FLIGHT'}
cur.execute(
'UPDATE accounts SET balance = balance - %s, version = version + 1 WHERE id = %s AND balance >= %s RETURNING balance, version',
(amount, account_id, amount),
)
row = cur.fetchone()
if row is None:
raise ValueError('Insufficient funds or missing account')
return {'status': 'COMMITTED', 'balance': row[0], 'version': row[1]}Atomic Conditional Debit & Idempotency Gate
1.Stage Objective: Claim idempotency and execute conditional balance debit in one ACID transaction.
2.Level (Intermediate): def execute_atomic_debit(cur, account_id: str, amount: int, idem_key: str, payload_hash: str) -> dict:
3.Operational Invariant: Claim idempotency and execute conditional balance debit in one ACID transaction.
Caption: Claim idempotency and execute conditional balance debit in one ACID transaction.
Sharding & Hot-Spot Mitigation.
Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.
Sharding & Hot-Spot Mitigation
1.Stage Objective: Prevent single-partition bottlenecks at peak load.
2.Level (Intermediate): Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.
3.Operational Invariant: Prevent single-partition bottlenecks at peak load.
Caption: Prevent single-partition bottlenecks at peak load.
Failure Modes & Graceful Degradation.
Apply circuit breakers, fallback caches, and dead-letter quarantine queues.
Failure Modes & Graceful Degradation
1.Stage Objective: Keep critical paths healthy during partial outages.
2.Level (Advanced): Apply circuit breakers, fallback caches, and dead-letter quarantine queues.
3.Operational Invariant: Keep critical paths healthy during partial outages.
Caption: Keep critical paths healthy during partial outages.
Consistency, Idempotency & Recovery.
Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.
Consistency, Idempotency & Recovery
1.Stage Objective: Guarantee correctness across retries and node crashes.
2.Level (Advanced): Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.
3.Operational Invariant: Guarantee correctness across retries and node crashes.
Caption: Guarantee correctness across retries and node crashes.
Telemetry, Circuit Breakers & Failure Modes.
Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants degrade.
Production Guardrails
1.Stage Objective: Monitor tail latency, error budgets, and invariant violations in production.
2.Level (Advanced): Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants de
3.Operational Invariant: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.
Caption: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.
Practice: Make the trade-off yourself.
Why is `UPDATE accounts SET balance = balance - amt WHERE id = ? AND balance >= amt` immune to TOCTOU?