Outbox Table Partitioning: Scaling Event Delivery Without Locking

Scenario context

Partition your outbox table to avoid write contention and speed up event reads—just remember to keep partitions balanced and keep your cleanup job in sync! 🚀 You’re building a microservice that writes domain events into a single outbox table as part of a transaction. Under load, every write hits the same table, causing lock contention, slow commits, and a backlog of events for the publisher. The question is: **how…

This walkthrough focuses on The Bottleneck Problem.

Before making a design choice, clarify:

• You’re building a microservice that writes domain events into a single outbox table as part of a transaction.
• Under load, every write hits the same table, causing lock contention, slow commits, and a backlog of events for the publisher.
• The question is: how do you keep the outbox write fast while still guaranteeing reliable event delivery?

Concepts in play: event-driven-architecture, transaction-isolation-distributed-acid, database-sharding

Stage 1 — Partitioning Strategy

• Use table partitioning (PostgreSQL, MySQL, or a sharded NoSQL table) to spread writes across multiple physical segments.
• Common strategies: Example: PostgreSQL hash partition

CREATE TABLE outbox (
  id          BIGSERIAL PRIMARY KEY,
  tenant_id   BIGINT NOT NULL,
  event_type  TEXT NOT NULL,
  payload     JSONB NOT NULL,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
) PARTITION BY HASH (tenant_id);

CREATE TABLE outbox_p0 PARTITION OF outbox FOR VALUES WITH (MODULUS 4, REMAINDER 0);
CREATE TABLE outbox_p1 PARTITION OF outbox FOR VALUES WITH (MODULUS 4, REMAINDER 1);
-- … up to p3

Observable outcome: Common strategies: Example: PostgreSQL hash partition

Stage 2 — Architecture Kit Diagram

• { "version": 1, "title": "Partitioned Outbox with Pull‑Based Publisher", "description": "Writes go to a hash‑partitioned outbox; a polling service reads from each partition in parallel.", "view": "data-flow", "pattern":…

Observable outcome: { "version": 1, "title": "Partitioned Outbox with Pull‑Based Publisher", "description": "Writes go to a hash‑partitioned outbox; a polling service reads from each partition in parallel.", "view": "data-flow", "pattern":…

Stage 3 — Trade‑offs & Gotchas

Observable outcome: You can explain how “Trade‑offs & Gotchas” changes the architecture or implementation path.

Stage 4 — Quick Checklist

• Choose a stable partition key that distributes writes evenly.
• Monitor partition sizes ; add or drop partitions as needed.
• Implement idempotent reads in the publisher to handle duplicate reads from partitions.
• Schedule cleanup (e.g., DROP TABLE outbox p0 for old data) after confirming delivery.

Observable outcome: Schedule cleanup (e.g., DROP TABLE outbox p0 for old data) after confirming delivery.

End-to-end outcome

You’re building a microservice that writes domain events into a single outbox table as part of a transaction.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.

• Use table partitioning (PostgreSQL, MySQL, or a sharded NoSQL table) to spread writes across multiple physical segments.
• { "version": 1, "title": "Partitioned Outbox with Pull‑Based Publisher", "description": "Writes go to a hash‑partitioned outbox; a polling service reads from each partition in parallel.", "view": "data-flow", "pattern":…
• Apply database-sharding to Trade‑offs & Gotchas.
• Choose a stable partition key that distributes writes evenly.