Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Designing Idempotent APIs to
Prevent Duplicate Billing.

Your billing-service (Java 21 + PostgreSQL 16 + Redis) processes 2,500 POST /v1/charges requests/sec. On flaky 4G mobile networks, a payment succeeds at the card processor (taking 2.1s), but the HTTP response drops before reaching the client. The mobile SDK and Envoy ingress automatically retry POST /v1/charges up to 3 times. Because billing-service has no idempotency-key enforcement, 0.4% of transactions (10/sec) are double- or triple-charged, causing $180,000/day in chargebacks and support tickets. Design a strictly consistent, crash-safe idempotency layer that guarantees at-most-once charging even under concurrent identical retries, server crashes mid-flight, and payload-mismatch bugs.
Start the journey5 stages
Loading diagram…
Designing Idempotent APIs to Prevent Duplicate Billing architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your `billing-service` (Java 21 + PostgreSQL 16 + Redis) processes 2,500 `POST /v1/charges` requests/sec. On flaky 4G mobile networks, a payment succeeds at the card processor (taking 2.1s), but the HTTP response drops before reaching the client. The mobile SDK and Envoy ingress automatically retry `POST /v1/charges` up to 3 times. Because `billing-service` has no idempotency-key enforcement, **0.4% of transactions (10/sec)** are double- or triple-charged, causing $180,000/day in chargebacks and support tickets. Design a strictly consistent, crash-safe idempotency layer that guarantees at-most-once charging even under concurrent identical retries, server crashes mid-flight, and payload-mismatch bugs.

From patterns to decisions.

Your `billing-service` (Java 21 + PostgreSQL 16 + Redis) processes 2,500 `POST /v1/charges` requests/sec. On flaky 4G mobile networks, a payment succeeds at the card processor (taking 2.1s), but the HTTP response drops before reaching the client. The mobile SDK and Envoy ingress automatically retry `POST /v1/charges` up to 3 times. Because `billing-service` has no idempotency-key enforcement, **0.4% of transactions (10/sec)** are double- or triple-charged, causing $180,000/day in chargebacks and support tickets. Design a strictly consistent, crash-safe idempotency layer that guarantees at-most-once charging even under concurrent identical retries, server crashes mid-flight, and payload-mismatch bugs.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Implementation & Guardrails (1).

Concurrent retries arriving milliseconds apart pass SELECT simultaneously (TOCTOU race), and holding a DB transaction open during the 2.1s PSP call exhausts the pool.

ChoiceWhat it buysWhat it costs
factorHolding DB Tx vs. Two-Phase State Machine; pro: Single Tx is simpler code.; con: Needs 5,250+ DB connections at 2,500 RPS—crashes PostgreSQL.

Step 1

Trace two retries arriving 5ms apart both seeing 0 rows on SELECT.

Calculate DB pool exhaustion

2,500 RPS * 2.1s = 5,250 concurrent DB connections needed if held open.

Alternative 1

Distributed Redis lock (SET NX EX) + PostgreSQL unique constraint.

name

Connection Pool Exhaustion; description: HikariCP max=50 saturates in 20ms during PSP slowdowns.; mitigation: Release DB connection between Tx 1 (claim) and Tx 2 (finalize).

-- Anti-pattern: TOCTOU race + 2.1s connection hold
BEGIN;
SELECT * FROM charges WHERE idempotency_key = $1; -- Both retries see 0 rows!
-- 2.1s external HTTP call...
COMMIT;
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Claim IN_PROGRESS atomically in Tx.

Claim IN_PROGRESS atomically in Tx 1, call PSP with the forwarded key, and persist COMPLETED + response JSON in Tx 2.

ChoiceWhat it buysWhat it costs
factorHourly Partitioning; pro: Drops 108 GB/day with < 10 KB WAL.; con: Requires checking current + previous hour partition if key spans an hour boundary.

Step 1

Partition idempotency_keys hourly for O(1) 24h expiry.

Step 2

Claim via INSERT ... ON CONFLICT DO NOTHING.

Step 3

Compare SHA-256 request hash; return 422 on mismatch, 409 on IN_PROGRESS, cached JSON on COMPLETED.

Alternative 1

Use UUIDv7 keys so created_at is embedded directly in the key for single-partition routing.

name

Hour-Boundary Split; description: A retry arriving at 01:00:01 after an 00:59:59 claim misses the new hour's partition if created_at is part of the PK.; mitigation: Use UUIDv7 or query created_at >= NOW() - INTERVAL '24 hours' with a global/per-partition lookup.

ClaimResult claim = repo.claimKey(merchantId, key, sha256(body));
if (claim.isDuplicate()) {
    if (!Arrays.equals(claim.hash(), sha256(body))) throw new UnprocessableEntityException();
    if (claim.status() == IN_PROGRESS) throw new ConflictException("Retry-After: 1");
    return ResponseEntity.status(claim.httpStatus()).body(claim.responseBody());
}
// Execute PSP call with key, then Tx 2 finalize()
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Cache terminal business outcomes (200.

Cache terminal business outcomes (200, 402), but clear/mark RETRYABLE on transient 503/network timeouts (after verifying the PSP didn't charge) so retries can succeed.

ChoiceWhat it buysWhat it costs
factorReleasing Key on Transient 503; pro: Allows client's automatic retry to succeed.; con: Must only release if PSP explicitly guarantees non-processing (otherwise risk double charge).

Step 1

Cache 200 OK and 402 Card Declined permanently for 24h.

Step 2

On 5xx or timeout, keep IN_PROGRESS until reaper verifies PSP state, or delete key if PSP returned a clean pre-auth 503.

Alternative 1

Require client to mint a new Idempotency-Key on 503 (poor UX for auto-retries).

name

Ambiguous Timeout Released Prematurely; description: Releasing key on read timeout lets retry double-charge.; mitigation: Never release on read timeout; reconcile with PSP first.

if (pspResponse.isTerminal()) {
    repo.complete(key, pspResponse.status(), pspResponse.body());
} else if (pspResponse.isGuaranteedNotProcessed()) {
    repo.releaseKey(key);
}
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Put a Redis read-through cache.

Put a Redis read-through cache in front of PostgreSQL for COMPLETED keys so 50,000 duplicate replays never touch PostgreSQL.

ChoiceWhat it buysWhat it costs
factorRedis Read-Through + PostgreSQL Authority; pro: Absorbs replay storms in < 0.5 ms without losing ACID safety.; con: Redis is strictly a performance cache; misses still hit PG.

Check Redis `GET idem

{merchant}:{key} first; if COMPLETED`, verify hash and return in < 0.5 ms.

Step 2

On Redis miss, fall through to authoritative PostgreSQL INSERT ... ON CONFLICT.

Alternative 1

Rate-limit identical Idempotency-Key replays at Envoy.

name

Stale Cache; description: Caching IN_PROGRESS in Redis forever.; mitigation: Only cache terminal COMPLETED responses in Redis.

cached = redis.get(f"idem:{merchant}:{key}")
if cached:
    return verify_hash_and_return(cached, body_hash)
# Fallback to PostgreSQL authoritative claim
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Reject with 400 Bad Request.

Reject with 400 Bad Request; generating a key server-side defeats idempotency because each retry would generate a brand-new key.

ChoiceWhat it buysWhat it costs
factorStrict 400 Rejection vs. Optional Header; pro: Forces every client integration to be retry-safe before going live.; con: Requires a brief grace/warning period for legacy integrations.

Step 1

Enforce Idempotency-Key presence at the API Gateway / controller filter.

Step 2

Return 400 Bad Request with clear documentation link.

Alternative 1

Derive deterministic key (order_id + attempt_id) only if the domain already has a unique order_id.

name

Server-Generated UUID; description: Provides false confidence while still double-charging on retries.; mitigation: Never generate idempotency keys server-side for POST.

if (idempotencyKey == null || !UUID_REGEX.matcher(idempotencyKey).matches()) {
    return ResponseEntity.badRequest().body("Missing or invalid Idempotency-Key");
}

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Require client-generated Idempotency-Key (UUIDv4) persisted with a unique DB constraint (merchant_id, idempotency_key) + payload SHA-256 hash inside the initial transaction, caching the exact response for safe replays.
  • 02Trade-off tip: factor: Releasing Key on Transient 503; pro: Allows client's automatic retry to succeed.; con: Must only release if PSP explicitly guarantees non-processing (otherwise risk double charge).
  • 03Watch out for: name: Stale Cache; description: Caching IN_PROGRESS in Redis forever.; mitigation: Only cache terminal COMPLETED responses in Redis.
Behavioral Progression · Idempotent Apis Preventing Duplicate Charges

Idempotent Apis Preventing Duplicate Charges — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Require client-generated Idempotency-Key (UUIDv4) persisted with a unique DB constraint (merchantid, idempotencykey) + payload SHA-256 hash inside the initial transaction, caching the exact response for safe replays.

Synthesis / One connected design

Putting Designing Idempotent APIs to Prevent Duplicate Billing Together

Require client-generated `Idempotency-Key` (UUIDv4) persisted with a unique DB constraint `(merchant_id, idempotency_key)` + payload SHA-256 hash inside the initial transaction, caching the exact response for safe replays.

A candidate architecture, not a universal answer

Putting Designing Idempotent APIs to Prevent Duplicate Billing Together

Step 1
Step 2
Step 3
  1. Partition idempotency_keys hourly for O(1) 24h expiry.
  2. Claim via INSERT ... ON CONFLICT DO NOTHING.
  3. Compare SHA-256 request hash; return 422 on mismatch, 409 on IN_PROGRESS, cached JSON on COMPLETED.
Put a Redis read-through cache in front of PostgreSQL for `COMPLETED` keys so 50,000 duplicate replays never touch PostgreSQL.
Interview Readiness Check

Put a Redis read-through cache in front of PostgreSQL for COMPLETED keys so 50,000 duplicate replays never touch PostgreSQL.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Designing Idempotent APIs to Prevent Duplicate Billing

Require client-generated Idempotency-Key (UUIDv4) persisted with a unique DB constraint (merchant_id, idempotency_key) + payload SHA-256 hash inside the initial transaction, caching the exact response for safe replays.