System Design HLD Example: Payment Processing Platform

Scenario context

Payment systems optimize for correctness first, then throughput. This guide covers idempotency, double-entry ledgers, and reconciliation.

Stripe processes over 250 million API requests per day, and every single payment must be idempotent: a user clicking "Pay" twice must never produce two charges, even when the first HTTP call timed out and no confirmation reached the client. The Knight Capital incident of 2012 — w…

This walkthrough focuses on Payment Processing: Use Cases & Requirements.

Before making a design choice, clarify:

• A robust payment platform must handle several distinct user journeys and operational requirements.
• In an interview, defining these boundaries early is critical.
• Functional Requirements - Idempotent Authorization : Ensure that a payment request is processed exactly once, regardless of retries.

Concepts in play: system-design-hld-architectures, caching, consistency-replication, jvm-internals-and-garbage-collection

Stage 1 — Basics: Baseline Architecture

• At its core, a payment system is a coordinator between three entities: the User (Client) , the Internal Ledger (Source of Truth) , and the External Payment Service Provider (PSP) .
• The baseline architecture consists of: 1.
• Payment API : The entry point that receives requests and orchestrates the flow.
• Idempotency Store : A fast, key-value store (like Redis) to track request keys.

Observable outcome: Idempotency Store : A fast, key-value store (like Redis) to track request keys.

Stage 2 — Mechanics: Key Logic

• The most critical logic in a payment system is the Persist-Before-Call pattern combined with Double-Entry Bookkeeping .
• The Persist-Before-Call Pattern To prevent lost payments, you must record the intent to pay in your database before calling the external PSP.
• This ensures that if the call to the PSP times out or the server crashes, you have a record to reconcile against later.
• Double-Entry Bookkeeping Money is never simply "updated" in a row.

sequenceDiagram
    participant U as User
    participant API as Payment API
    participant Redis as Idempotency Store
    participant DB as Ledger DB
    participant PSP as Payment Provider
    U->>API: POST /payments with Idempotency-Key
    API->>Redis: Check existing key
    Redis-->>API: Not found — proceed
    API->>DB: Write PENDING intent
    DB-->>API: Intent persisted
    API->>PSP: Authorize charge with intent_id
    PSP-->>API: Authorization success
    API->>DB: Update status to AUTHORIZED
    API->>Redis: Cache response with 24h TTL
    API-->>U: Payment confirmed

Observable outcome: Double-Entry Bookkeeping Money is never simply "updated" in a row.

Stage 3 — Estimations & Design Goals

• Before diving into the high-level design, let's establish the scale.
• • Throughput : 1,000 Transactions Per Second (TPS) average, 5,000 TPS peak (e.g., Black Friday).
• • Storage : Each transaction record is ~2KB.
• At 1,000 TPS, that's 2MB/sec, or ~172GB/day.

Observable outcome: At 1,000 TPS, that's 2MB/sec, or ~172GB/day.

Stage 4 — High-Level Design

• The following diagram illustrates the orchestrated flow of a payment transaction.
• The flow enforces the Persist-Before-Call pattern: the Payment API writes a PENDING intent record to the PostgreSQL ledger (step 2) before calling the external PSP (step 3).
• This ensures that even if the network drops between steps 3 and 4, the system has a record to reconcile against.
• The idempotency check in Redis (step 1) prevents a duplicate upstream call if the client retries with the same key.

graph TD
    User[Customer/App] -->|POST /payments| API[Payment API]
    API -->|1. Check/Lock Key| Redis[(Idempotency Redis)]
    API -->|2. Create Intent| Ledger[(PostgreSQL Ledger)]
    API -->|3. Authorize| PSP[Payment Provider - Stripe/Adyen]
    PSP -->|4. Response| API
    API -->|5. Update Status| Ledger
    API -->|6. Cache Result| Redis
    Ledger -->|7. CDC Event| Kafka{Kafka}
    Kafka -->|8. Notify| Webhook[Webhook Service]
    Webhook -->|9. Callback| Merchant[Merchant Server]
    Recon[Reconciliation Job] -->|Daily Sync| PSP
    Recon -->|Verify| Ledger

Observable outcome: The idempotency check in Redis (step 1) prevents a duplicate upstream call if the client retries with the same key.

Stage 5 — Deep Dive: Double-Entry Bookkeeping, Idempotency Key Lifecycle, and the Reconciliation Engine

• Three mechanisms define correctness in a payment system: double-entry bookkeeping that makes every financial state change auditable, idempotency that prevents duplicate charges from network retries, and reconciliation t…
• Internals: Double-Entry Bookkeeping and Why Integer Cents Matter In a double-entry ledger, money is never simply "updated in a row." Every movement of value produces at minimum two ledger entries: a debit from one accou…
• When a customer pays $100 to a merchant, the system writes: DEBIT $100 from the customer's account, CREDIT $100 to the merchant's escrow account.
• The fundamental accounting invariant is that the sum of all debits must equal the sum of all credits across the entire ledger at all times.

flowchart TD
    A["Request Received with Idempotency-Key"] --> B{"Key Exists in Redis?"}
    B -->|Yes| C["Return Cached Response Immediately"]
    B -->|No| D["Acquire Distributed Lock — 30s TTL"]
    D --> E["Write PENDING Intent to Ledger"]
    E --> F["Call External PSP"]
    F --> G{"PSP Response Received?"}
    G -->|Success| H["Update Ledger to AUTHORIZED"]
    G -->|Failure| I["Update Ledger to FAILED"]
    H --> J["Cache Success Response — 24h TTL"]
    I --> J
    J --> K["Release Lock and Return to Caller"]

Observable outcome: The fundamental accounting invariant is that the sum of all debits must equal the sum of all credits across the entire ledger at all times.

Stage 6 — Real-World Payment Architectures: Stripe, PayPal, and Square

• Stripe built its idempotency key system as a first-class API primitive from day one.
• Every write endpoint in the Stripe API accepts an Idempotency-Key header.
• Stripe stores the idempotency key, the request fingerprint (URL + body hash), and the response body in a dedicated idempotency database with a 24-hour retention window.
• Critically, Stripe validates that the same idempotency key is not used with a different request body — preventing a class of bugs where a client reuses a key across conceptually different operations.

Observable outcome: Critically, Stripe validates that the same idempotency key is not used with a different request body — preventing a class of bugs where a client reuses a key across conceptually different operations.

End-to-end outcome

A robust payment platform must handle several distinct user journeys and operational requirements.

Alternatives considered

Decision Guide: Choosing the Right Payment Guarantee Level for Each Operation

Decision Guide: Choosing the Right Payment Guarantee Level for Each Operation

Failure and degraded modes

Trade-offs and Failure Modes in Payment System Design

• Detection: The Double-Charge Failure Mode: The API calls the PSP, the PSP charges the card and returns success, but the network drops before the response reaches the API.
• User or operator impact: The Double-Charge Failure Mode: The API calls the PSP, the PSP charges the card and returns success, but the network drops before the response reaches the API.

Lessons Learned from Payment System Failures

• Detection: The Knight Capital Incident: No Idempotency Guard on Algorithmic Orders.
• User or operator impact: The Knight Capital Incident: No Idempotency Guard on Algorithmic Orders.