Design the distributed transaction architecture for an e-commerce checkout flow spanning OrderService (PostgreSQL), InventoryService (PostgreSQL), and PaymentService (Stripe + Ledger) at 2,500 checkouts/sec. Compare 2PC (XA) against the Saga Pattern and implement a crash-safe Orchestrated Saga.
From patterns to decisions.
Design the distributed transaction architecture for an e-commerce checkout flow spanning OrderService (PostgreSQL), InventoryService (PostgreSQL), and PaymentService (Stripe + Ledger) at 2,500 checkouts/sec. Compare 2PC (XA) against the Saga Pattern and implement a crash-safe Orchestrated Saga.
In 2PC, participants that vote.
In 2PC, participants that vote YES in Phase 1 enter an IN_DOUBT state holding exclusive row locks until the coordinator sends COMMIT or ABORT. External REST APIs lack XA prepare/commit hooks.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | 2PC provides strict serializable isolation automatically, whereas Sagas require explicit semantic locks. |
Step 1
Trace Phase 1 (Prepare) and Phase 2 (Commit/Abort) message sequence.
Step 2
Show how a coordinator crash after Phase 1 wedges participant locks.
Step 3
Explain why holding DB locks during a 200 ms Stripe call collapses throughput.
Alternative 1
Paxos/Raft-replicated 2PC (used inside Google Spanner) for homogeneous internal storage.
Failure Mode 1
Participant lock starvation during coordinator outage or slow downstream network.
// Why 2PC stalls: DB connection & row lock held across external HTTP RTT
@Transactional
public void brokenTwoPhaseCheckout(String sku, int qty, String cardToken) {
inventoryRepo.lockAndDecrement(sku, qty); // Acquires exclusive PostgreSQL row lock
stripeClient.charge(cardToken); // 200ms network call -> caps SKU at ~5 TPS!
}
Implementation & Guardrails (2).
Java state-machine orchestrator persisting Saga state transitions alongside outbox commands, classifying steps into compensatable, pivot, and retryable.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Row-level `findForUpdate` on `sagaId` serializes concurrent callbacks for the same checkout without blocking other checkouts. |
Step 1
Persist SagaState (STARTED, INVENTORY_RESERVED, PAYMENT_CHARGED, CONFIRMED, COMPENSATING, FAILED) in local DB.
Step 2
On failure at ChargePayment (Pivot), transition to COMPENSATING and run releaseInventory -> cancelOrder.
Step 3
On success at Pivot, transition to PAYMENT_CHARGED and retry confirmOrder until completion.
Alternative 1
Temporal.io workflow engine managing durable execution and timers automatically.
Failure Mode 1
Publishing to Kafka outside the DB transaction before sagaRepo.save commits causes ghost events on rollback.
public enum SagaStatus { STARTED, INVENTORY_RESERVED, PAYMENT_CHARGED, CONFIRMED, COMPENSATING, FAILED }
public void handleStepResult(String sagaId, String step, boolean success) {
SagaRecord saga = sagaRepo.findForUpdate(sagaId);
switch (saga.status()) {
case STARTED -> {
if (step.equals("RESERVE_INVENTORY") && success) {
saga.transitionTo(SagaStatus.INVENTORY_RESERVED);
outbox.enqueue(sagaId, "CHARGE_PAYMENT", saga.paymentCmd());
} else {
saga.transitionTo(SagaStatus.FAILED);
outbox.enqueue(sagaId, "CANCEL_ORDER", saga.orderCmd());
}
}
case INVENTORY_RESERVED -> {
if (step.equals("CHARGE_PAYMENT") && success) { // Pivot succeeded
saga.transitionTo(SagaStatus.PAYMENT_CHARGED);
outbox.enqueue(sagaId, "CONFIRM_ORDER", saga.orderCmd());
} else { // Pivot failed -> backward compensation
saga.transitionTo(SagaStatus.COMPENSATING);
outbox.enqueue(sagaId, "RELEASE_INVENTORY", saga.invCmd());
}
}
case COMPENSATING -> {
if (step.equals("RELEASE_INVENTORY") && success) {
saga.transitionTo(SagaStatus.FAILED);
outbox.enqueue(sagaId, "CANCEL_ORDER", saga.orderCmd());
}
}
case PAYMENT_CHARGED -> {
if (step.equals("CONFIRM_ORDER") && success) {
saga.transitionTo(SagaStatus.CONFIRMED);
} else {
retryScheduler.scheduleWithBackoff(sagaId, "CONFIRM_ORDER");
}
}
}
}
Orchestration keeps business flow, timeout.
Orchestration keeps business flow, timeout sweeps, and reverse compensation order in one explicit state machine, whereas Choreography scatters workflow logic across 3+ services and risks cyclic dependencies.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Orchestrator can become a 'god service' if business logic (e.g., pricing rules) leaks into it instead of staying purely a workflow coordinator. |
Compare coupling
Choreography couples services to each other's domain events; Orchestrator depends on participants via commands.
Compare observability
Orchestrator has a single saga_state table to query stuck checkouts.
Step 3
Note that Choreography works well for 2-step fan-out, while Orchestration scales cleanly to 3+ steps with compensation.
Alternative 1
Hybrid: Orchestrator for core checkout, Choreography for post-order analytics/emails.
Failure Mode 1
In Choreography, adding a 4th step (FraudCheck) requires modifying event subscriptions across multiple existing services.
// Idempotent participant handler using unique (saga_id, step) constraint
public void onReserveInventory(String sagaId, String sku, int qty) {
if (idempotencyRepo.tryInsert(sagaId, "RESERVE_INVENTORY")) {
int updated = inventoryRepo.reserve(sku, qty);
outbox.publishReply(sagaId, "RESERVE_INVENTORY", updated > 0);
}
// Duplicate delivery is safely ignored
}
Out-of-order compensation is solved by.
Out-of-order compensation is solved by inserting a COMPENSATED tombstone in the idempotency log so late forward commands abort; Stripe 504s must be resolved via idempotent status polling before compensating.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Requires retaining `idempotency_log` entries for at least the max message retention window (e.g., 7 days). |
Step 1
On ReleaseInventory, if RESERVE is not in idempotency_log, insert (sagaId, 'RESERVE', 'ABORTED') so a late Reserve is a no-op.
Step 2
On Stripe 504 timeout, call GET /v1/payment_intents/{id} with the same Idempotency-Key.
Step 3
Never compensate a payment until Stripe confirms the intent is canceled or requires_payment_method.
Alternative 1
Fencing tokens / monotonic step sequence numbers on participant records.
Failure Mode 1
Compensating on a 504 timeout when Stripe actually charged the card results in a free order or double-charge.
// Out-of-order compensation guard
public void onReleaseInventory(String sagaId, String sku, int qty) {
if (!idempotencyRepo.tryInsert(sagaId, "RELEASE_INVENTORY")) return;
if (idempotencyRepo.markAbortedIfAbsent(sagaId, "RESERVE_INVENTORY")) {
// Reserve never arrived yet; tombstone prevents late Reserve from executing
return;
}
inventoryRepo.release(sku, qty);
}
Use Semantic Locks (status =.
Use Semantic Locks (status = PENDING_PAYMENT) on Order and separate available vs reserved columns on Inventory so fulfillment and customer views ignore in-flight records until confirmed.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Adds application-level state complexity compared to database-level `SERIALIZABLE` isolation. |
Step 1
Filter fulfillment/shipping queries with WHERE status = 'CONFIRMED'.
Step 2
Use available >= qty for new checkouts, isolating reserved stock until Saga completion.
Step 3
Expose an eventual-consistency PROCESSING UI state to the user during the ~250 ms Saga window.
Alternative 1
Read-your-own-writes session stickiness via CQRS read-model projection.
Failure Mode 1
Warehouse fulfillment jobs querying orders without status = 'CONFIRMED' ship unpaid items.
// Atomic semantic reservation: prevents overselling without cross-service locks
// SQL: UPDATE inventory SET available = available - :qty, reserved = reserved + :qty
// WHERE sku = :sku AND available >= :qty;
boolean reserved = inventoryRepo.atomicReserve(sku, qty) == 1;
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: 2PC guarantees atomic isolation (ACD) by holding locks across network RTTs, making it vulnerable to coordinator crashes; Sagas trade strict isolation for high availability and throughput using eventual consistency and compensations.
- 02Trade-off tip: Orchestrator can become a 'god service' if business logic (e.g., pricing rules) leaks into it instead of staying purely a workflow coordinator.
- 03Watch out for: Compensating on a 504 timeout when Stripe actually charged the card results in a free order or double-charge.
Two Phase Commit Vs Saga Pattern — Normal → Load / Fault → Recovery
2PC guarantees atomic isolation (ACD) by holding locks across network RTTs, making it vulnerable to coordinator crashes; Sagas trade strict isolation for high availability and throughput using eventual consistency and compensations.
Putting Two-Phase Commit (2PC) vs. the Saga Pattern Together
2PC guarantees atomic isolation (ACD) by holding locks across network RTTs, making it vulnerable to coordinator crashes; Sagas trade strict isolation for high availability and throughput using eventual consistency and compensations.
Putting Two-Phase Commit (2PC) vs. the Saga Pattern Together
- Persist SagaState (
STARTED,INVENTORY_RESERVED,PAYMENT_CHARGED,CONFIRMED,COMPENSATING,FAILED) in local DB. - On failure at
ChargePayment(Pivot), transition toCOMPENSATINGand runreleaseInventory->cancelOrder. - On success at Pivot, transition to
PAYMENT_CHARGEDand retryconfirmOrderuntil completion.
Interview Readiness Check
Out-of-order compensation is solved by inserting a COMPENSATED tombstone in the idempotency log so late forward commands abort; Stripe 504s must be resolved via idempotent status polling before compensating.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is Two-Phase Commit the right default for a production team?
Two-Phase Commit (2PC) vs. the Saga Pattern
2PC guarantees atomic isolation (ACD) by holding locks across network RTTs, making it vulnerable to coordinator crashes; Sagas trade strict isolation for high availability and throughput using eventual consistency and compensations.
