Your `billing-service` (Java 21 + PostgreSQL 16 + Redis) processes 2,500 `POST /v1/charges` requests/sec. On flaky 4G mobile networks, a payment succeeds at the card processor (taking 2.1s), but the HTTP response drops before reaching the client. The mobile SDK and Envoy ingress automatically retry `POST /v1/charges` up to 3 times. Because `billing-service` has no idempotency-key enforcement, **0.4% of transactions (10/sec)** are double- or triple-charged, causing $180,000/day in chargebacks and support tickets. Design a strictly consistent, crash-safe idempotency layer that guarantees at-most-once charging even under concurrent identical retries, server crashes mid-flight, and payload-mismatch bugs.
From patterns to decisions.
Your `billing-service` (Java 21 + PostgreSQL 16 + Redis) processes 2,500 `POST /v1/charges` requests/sec. On flaky 4G mobile networks, a payment succeeds at the card processor (taking 2.1s), but the HTTP response drops before reaching the client. The mobile SDK and Envoy ingress automatically retry `POST /v1/charges` up to 3 times. Because `billing-service` has no idempotency-key enforcement, **0.4% of transactions (10/sec)** are double- or triple-charged, causing $180,000/day in chargebacks and support tickets. Design a strictly consistent, crash-safe idempotency layer that guarantees at-most-once charging even under concurrent identical retries, server crashes mid-flight, and payload-mismatch bugs.
Implementation & Guardrails (1).
Concurrent retries arriving milliseconds apart pass SELECT simultaneously (TOCTOU race), and holding a DB transaction open during the 2.1s PSP call exhausts the pool.
| Choice | What it buys | What it costs |
|---|---|---|
| factor | Holding DB Tx vs. Two-Phase State Machine; pro: Single Tx is simpler code.; con: Needs 5,250+ DB connections at 2,500 RPS—crashes PostgreSQL. |
Step 1
Trace two retries arriving 5ms apart both seeing 0 rows on SELECT.
Calculate DB pool exhaustion
2,500 RPS * 2.1s = 5,250 concurrent DB connections needed if held open.
Alternative 1
Distributed Redis lock (SET NX EX) + PostgreSQL unique constraint.
name
Connection Pool Exhaustion; description: HikariCP max=50 saturates in 20ms during PSP slowdowns.; mitigation: Release DB connection between Tx 1 (claim) and Tx 2 (finalize).
-- Anti-pattern: TOCTOU race + 2.1s connection hold
BEGIN;
SELECT * FROM charges WHERE idempotency_key = $1; -- Both retries see 0 rows!
-- 2.1s external HTTP call...
COMMIT;
Claim IN_PROGRESS atomically in Tx.
Claim IN_PROGRESS atomically in Tx 1, call PSP with the forwarded key, and persist COMPLETED + response JSON in Tx 2.
| Choice | What it buys | What it costs |
|---|---|---|
| factor | Hourly Partitioning; pro: Drops 108 GB/day with < 10 KB WAL.; con: Requires checking current + previous hour partition if key spans an hour boundary. |
Step 1
Partition idempotency_keys hourly for O(1) 24h expiry.
Step 2
Claim via INSERT ... ON CONFLICT DO NOTHING.
Step 3
Compare SHA-256 request hash; return 422 on mismatch, 409 on IN_PROGRESS, cached JSON on COMPLETED.
Alternative 1
Use UUIDv7 keys so created_at is embedded directly in the key for single-partition routing.
name
Hour-Boundary Split; description: A retry arriving at 01:00:01 after an 00:59:59 claim misses the new hour's partition if created_at is part of the PK.; mitigation: Use UUIDv7 or query created_at >= NOW() - INTERVAL '24 hours' with a global/per-partition lookup.
ClaimResult claim = repo.claimKey(merchantId, key, sha256(body));
if (claim.isDuplicate()) {
if (!Arrays.equals(claim.hash(), sha256(body))) throw new UnprocessableEntityException();
if (claim.status() == IN_PROGRESS) throw new ConflictException("Retry-After: 1");
return ResponseEntity.status(claim.httpStatus()).body(claim.responseBody());
}
// Execute PSP call with key, then Tx 2 finalize()
Cache terminal business outcomes (200.
Cache terminal business outcomes (200, 402), but clear/mark RETRYABLE on transient 503/network timeouts (after verifying the PSP didn't charge) so retries can succeed.
| Choice | What it buys | What it costs |
|---|---|---|
| factor | Releasing Key on Transient 503; pro: Allows client's automatic retry to succeed.; con: Must only release if PSP explicitly guarantees non-processing (otherwise risk double charge). |
Step 1
Cache 200 OK and 402 Card Declined permanently for 24h.
Step 2
On 5xx or timeout, keep IN_PROGRESS until reaper verifies PSP state, or delete key if PSP returned a clean pre-auth 503.
Alternative 1
Require client to mint a new Idempotency-Key on 503 (poor UX for auto-retries).
name
Ambiguous Timeout Released Prematurely; description: Releasing key on read timeout lets retry double-charge.; mitigation: Never release on read timeout; reconcile with PSP first.
if (pspResponse.isTerminal()) {
repo.complete(key, pspResponse.status(), pspResponse.body());
} else if (pspResponse.isGuaranteedNotProcessed()) {
repo.releaseKey(key);
}
Put a Redis read-through cache.
Put a Redis read-through cache in front of PostgreSQL for COMPLETED keys so 50,000 duplicate replays never touch PostgreSQL.
| Choice | What it buys | What it costs |
|---|---|---|
| factor | Redis Read-Through + PostgreSQL Authority; pro: Absorbs replay storms in < 0.5 ms without losing ACID safety.; con: Redis is strictly a performance cache; misses still hit PG. |
Check Redis `GET idem
{merchant}:{key} first; if COMPLETED`, verify hash and return in < 0.5 ms.
Step 2
On Redis miss, fall through to authoritative PostgreSQL INSERT ... ON CONFLICT.
Alternative 1
Rate-limit identical Idempotency-Key replays at Envoy.
name
Stale Cache; description: Caching IN_PROGRESS in Redis forever.; mitigation: Only cache terminal COMPLETED responses in Redis.
cached = redis.get(f"idem:{merchant}:{key}")
if cached:
return verify_hash_and_return(cached, body_hash)
# Fallback to PostgreSQL authoritative claim
Reject with 400 Bad Request.
Reject with 400 Bad Request; generating a key server-side defeats idempotency because each retry would generate a brand-new key.
| Choice | What it buys | What it costs |
|---|---|---|
| factor | Strict 400 Rejection vs. Optional Header; pro: Forces every client integration to be retry-safe before going live.; con: Requires a brief grace/warning period for legacy integrations. |
Step 1
Enforce Idempotency-Key presence at the API Gateway / controller filter.
Step 2
Return 400 Bad Request with clear documentation link.
Alternative 1
Derive deterministic key (order_id + attempt_id) only if the domain already has a unique order_id.
name
Server-Generated UUID; description: Provides false confidence while still double-charging on retries.; mitigation: Never generate idempotency keys server-side for POST.
if (idempotencyKey == null || !UUID_REGEX.matcher(idempotencyKey).matches()) {
return ResponseEntity.badRequest().body("Missing or invalid Idempotency-Key");
}
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Require client-generated
Idempotency-Key(UUIDv4) persisted with a unique DB constraint(merchant_id, idempotency_key)+ payload SHA-256 hash inside the initial transaction, caching the exact response for safe replays. - 02Trade-off tip: factor: Releasing Key on Transient 503; pro: Allows client's automatic retry to succeed.; con: Must only release if PSP explicitly guarantees non-processing (otherwise risk double charge).
- 03Watch out for: name: Stale Cache; description: Caching
IN_PROGRESSin Redis forever.; mitigation: Only cache terminalCOMPLETEDresponses in Redis.
Idempotent Apis Preventing Duplicate Charges — Normal → Load / Fault → Recovery
Require client-generated Idempotency-Key (UUIDv4) persisted with a unique DB constraint (merchantid, idempotencykey) + payload SHA-256 hash inside the initial transaction, caching the exact response for safe replays.
Putting Designing Idempotent APIs to Prevent Duplicate Billing Together
Require client-generated `Idempotency-Key` (UUIDv4) persisted with a unique DB constraint `(merchant_id, idempotency_key)` + payload SHA-256 hash inside the initial transaction, caching the exact response for safe replays.
Putting Designing Idempotent APIs to Prevent Duplicate Billing Together
- Partition
idempotency_keyshourly for O(1) 24h expiry. - Claim via
INSERT ... ON CONFLICT DO NOTHING. - Compare
SHA-256request hash; return422on mismatch,409onIN_PROGRESS, cached JSON onCOMPLETED.
Interview Readiness Check
Put a Redis read-through cache in front of PostgreSQL for COMPLETED keys so 50,000 duplicate replays never touch PostgreSQL.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Designing Idempotent APIs to Prevent Duplicate Billing
Require client-generated Idempotency-Key (UUIDv4) persisted with a unique DB constraint (merchant_id, idempotency_key) + payload SHA-256 hash inside the initial transaction, caching the exact response for safe replays.
