The Dual Write Problem: Why Two Writes Always Fail Eventually — and How to Fix It

Scenario context

Any service that writes to a database and publishes a message in the same logical operation has a dual write problem. try/catch retries don't fix it — they turn failures into duplicates. The Transactional Outbox pattern co-writes business data and a pending event into the same DB transaction, then a relay delivers to Kafka separately. At-least-once delivery, idempotent consumers — no ghost orders, no phantom f…

This walkthrough focuses on Why Every Instinctive Fix Makes the Problem Worse.

Before making a design choice, clarify:

• When engineers first encounter the dual write problem, there are three instinctive fixes.
• All three are wrong — not because of implementation details, but because of structural incompatibilities with the systems involved.
• Fix Attempt 1 — "Just Wrap It in a Try-Catch and Retry" The problem is that the catch block cannot distinguish which write failed.

Concepts in play: event-driven-architecture, caching, api-gateway, transaction-isolation-distributed-acid

Stage 1 — The Order That Silently Disappeared From Fulfillment

• It is 11:42 PM on Black Friday.
• An order service receives a checkout request, writes the new row to PostgreSQL — status PENDING , payment captured — and then calls kafkaTemplate.send("order-events", payload) .
• The Kafka broker is under peak load.
• A NetworkException is thrown, caught by catch (Exception e) { log.error(...); } , and silently swallowed.

Observable outcome: A NetworkException is thrown, caught by catch (Exception e) { log.error(...); } , and silently swallowed.

Stage 2 — Deep Dive: How the Transactional Outbox Achieves True Atomicity

• The Transactional Outbox pattern resolves the dual write problem by changing the question.
• Instead of "how do we atomically write to DB and Kafka?" — which is impossible — it asks: "what if both writes go to the database, and we let a separate process deliver to Kafka?" The answer is that you can write the bu…
• The two rows either commit together or roll back together.
• Kafka delivery becomes a separate, retryable, eventually-consistent operation.

Observable outcome: Kafka delivery becomes a separate, retryable, eventually-consistent operation.

Stage 3 — Visualizing the Dual Write Gap and the Outbox Solution End to End

• The diagram below contrasts the naive dual write (left path, failure gap visible) with the Transactional Outbox (right path, atomicity guaranteed through the shared DB transaction).
• The two subgraphs make the structural difference explicit.
• In the naive path, the commit and the Kafka send are sequential and independent — any exception between them creates a permanent gap.
• In the Outbox path, both writes happen inside one database transaction.

flowchart TD
    subgraph naive[Naive Dual Write - failure gap]
        N1[orderRepo.save] --> N2[COMMIT success]
        N2 --> N3[kafkaTemplate.send]
        N3 -->|NetworkException| N4[Event lost - gap opens]
    end

    subgraph outbox[Transactional Outbox - atomic]
        O1[BEGIN TX] --> O2[INSERT INTO orders]
        O2 --> O3[INSERT INTO outbox_events published=false]
        O3 --> O4[COMMIT - both rows durable]
        O4 --> O5[Outbox Relay - Scheduled 500ms]
        O5 --> O6[KafkaTemplate.send]
        O6 -->|success| O7[UPDATE outbox_events published=true]
        O6 -->|failure| O8[Row stays unpublished - retried next cycle]
    end

Observable outcome: In the Outbox path, both writes happen inside one database transaction.

Stage 4 — Where Dual Writes Appear Across Real Production Architecture Layers

• The dual write problem appears at every layer of a production system.
• Once you know the pattern, you see it everywhere.
• Order service (DB + Kafka): The canonical case.
• Save order → publish OrderPlaced .

Observable outcome: Save order → publish OrderPlaced .

Stage 5 — Choosing the Right Pattern for Your System's Constraints

• The decision tree below maps system constraints to the appropriate pattern.
• Start with the type of secondary system you are writing to, then apply the latency and complexity budget.
• The flowchart above guides the initial pattern selection.
• Use the decision table below to validate your choice against operational constraints.

flowchart TD
    A[Service writes to two systems] --> B{Secondary system type?}
    B -->|Message broker Kafka SQS RabbitMQ| C{Sub-100ms latency required?}
    C -->|Yes| D[CDC + Debezium Outbox Router - 50ms WAL-based]
    C -->|No| E[Transactional Outbox Polling Relay - 500ms to 2s]
    B -->|Cache Redis Memcached| F[Cache-Aside DELETE on write - read-through repopulates]
    B -->|Search Index Elasticsearch| G[CDC to ES Sink Connector]
    B -->|Another service DB or cross-service state| H{Audit trail or time-travel required?}
    H -->|Yes| I[Event Sourcing + CQRS]
    H -->|No| J[Transactional Outbox + Saga for compensation]

Observable outcome: Use the decision table below to validate your choice against operational constraints.

Stage 6 — Full Transactional Outbox Implementation in Spring Boot

• The following is a complete, production-ready implementation using Spring Data JPA for the database writes and Spring Kafka for event delivery.
• Each class maps directly to one of the three components described in the Deep Dive section.
• The OutboxEvent Entity — Staging Table for Pending Events The OrderService — Atomic Dual Write in a Single Transaction Both the business record and the outbox row are written inside one @Transactional boundary.
• If any step throws before the transaction commits, both rows roll back atomically.

@Entity
@Table(name = "outbox_events", indexes = {
    @Index(name = "idx_outbox_pending", columnList = "published, created_at")
})
public class OutboxEvent {

    @Id
    private String eventId;

    private String aggregateType;   // "Order", "Payment", "User"
    private String aggregateId;     // business entity ID — becomes Kafka message key
    private String eventType;       // "OrderPlaced", "PaymentCaptured"

    @Column(columnDefinition = "TEXT")
    private String payload;         // JSON-serialized domain event

    private Instant createdAt;
    private boolean published;

    protected OutboxEvent() {}

    public OutboxEvent(String eventId, String aggregateType, String aggregateId,
                       String eventType, String payload, Instant createdAt, boolean published) {
        this.eventId = eventId;
        this.aggregateType = aggregateType;
        this.aggregateId = aggregateId;
        this.eventType = eventType;
        this.payload = payload;
        this.createdAt = createdAt;
        this.published = published;
    }

    public String getEventId()     { return eventId; }
    public String getAggregateId() { return aggregateId; }
    public String getPayload()     { return payload; }
    public boolean isPublished()   { return published; }
    public void setPublished(boolean p) { this.published = p; }
}

@Service
@Transactional
public class OrderService {

    private final OrderRepository orderRepo;
    private final OutboxEventRepository outboxRepo;
    private final ObjectMapper mapper;

    public OrderService(OrderRepository orderRepo,
                        OutboxEventRepository outboxRepo,
                        ObjectMapper mapper) {
        this.orderRepo  = orderRepo;
        this.outboxRepo = outboxRepo;
        this.mapper     = mapper;
    }

    public Order placeOrder(PlaceOrderRequest req) throws JsonProcessingException {
        // Step 1: save business entity
        Order order = new Order(
            req.getCustomerId(),
            req.getItems(),
            OrderStatus.PENDING
        );
        orderRepo.save(order);

        // Step 2: write outbox event — SAME transaction as step 1
        String payload = mapper.writeValueAsString(
            new OrderPlacedEvent(order.getId(), req.getCustomerId(), order.getTotalAmount())
        );
        OutboxEvent event = new OutboxEvent(
            UUID.randomUUID().toString(),   // unique event_id — Kafka key for dedup
            "Order",
            order.getId().toString(),
            "OrderPlaced",
            payload,
            Instant.now(),
            false
        );
        outboxRepo.save(event);

        return order;
        // @Transactional commits both rows here — or rolls both back on any exception
    }
}

Observable outcome: If any step throws before the transaction commits, both rows roll back atomically.

End-to-end outcome

When engineers first encounter the dual write problem, there are three instinctive fixes.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

Before exploring solutions, it is essential to see all three failure modes. Engineers who only know Scenario A are surprised by B and C in production.

• Detection: Before exploring solutions, it is essential to see all three failure modes.
• User or operator impact: Before exploring solutions, it is essential to see all three failure modes.

Scenario A — The Ghost Order: DB Succeeds, Kafka Fails

• Detection: The PostgreSQL transaction commits successfully.
• User or operator impact: The PostgreSQL transaction commits successfully.

Scenario B — The Phantom Fulfillment: Kafka Succeeds, DB Fails

• Detection: The failure can run in the opposite direction, and the consequences are worse.
• User or operator impact: The failure can run in the opposite direction, and the consequences are worse.

Scenario C — The Stale Cache: DB Succeeds, Redis Fails

• Detection: Not every dual write involves Kafka.
• User or operator impact: Not every dual write involves Kafka.

Trade-offs and Failure Modes Across Outbox, CDC, and Event Sourcing

• Detection: No solution is universally optimal.
• User or operator impact: No solution is universally optimal.

Hard Lessons from Dual Write Failures in Production

• Detection: Every service that writes to DB and publishes an event has a dual write problem.
• User or operator impact: Every service that writes to DB and publishes an event has a dual write problem.