Low-Level Design: Building a Thread-Safe In-Memory Pub-Sub Queue

Scenario context

Building an in-memory message queue is a premier LLD problem testing multithreaded design. By combining the observer pattern with Java reentrant locks, conditions, and concurrent atomic offsets, we implement a thread-safe broker that decouples producers and consumers.

Imagine you are building an in-memory logging and analytics processing engine for a microservice. Different parts of the application (e.g., HT…

This walkthrough focuses on Design Challenge: Thread Contention in Event-Driven Systems.

Before making a design choice, clarify:

• Imagine you are building an in-memory logging and analytics processing engine for a microservice.
• Different parts of the application (e.g., HTTP request loggers, billing events, security auditors) generate messages that must be processed asynchronously by multiple worker threads.
• A naive observer pattern implementation couples the publisher and subscriber.

Concepts in play: low-level-design-systems, stack-and-queue-data-structures, jvm-internals-and-garbage-collection, message-queues

Stage 1 — Scope: System Actors and Core Requirements

• Before implementing the class model, we establish the requirements for our in-memory queue and define how actors interact.
• System Actors Producer: Represents the client thread generating messages.
• It publishes events to specific topics.
• Consumer: An independent execution thread that continuously polls messages from specific topics and partitions.

Observable outcome: Consumer: An independent execution thread that continuously polls messages from specific topics and partitions.

Stage 2 — Architectural Blueprint: Class Diagram

• The class diagram below outlines the structural composition of our in-memory queue: This class diagram details the relationships between the core components of our concurrent queue broker.
• The QueueBroker manages multiple Topic objects.
• Each Topic contains multiple TopicPartition structures.
• Each partition encapsulates a thread-safe array of Message entities, utilizing a ReentrantLock and a Condition variable to coordinate producers and consumers safely.

classDiagram
    QueueBroker "1" *-- "*" Topic : manages
    Topic "1" *-- "*" TopicPartition : contains
    TopicPartition "1" *-- "*" Message : holds
    ConsumerGroup "1" *-- "*" ConsumerInstance : coordinates
    ConsumerInstance --> QueueBroker : polls from
    class TopicPartition {
        -List~Message~ messages
        -ReentrantLock lock
        -Condition hasMessages
        +append(Message msg) void
        +read(int offset) Message
    }
    class ConsumerInstance {
        -String id
        -ConsumerGroup group
        +poll() void
    }

Observable outcome: Each partition encapsulates a thread-safe array of Message entities, utilizing a ReentrantLock and a Condition variable to coordinate producers and consumers safely.

Stage 3 — Core Mechanics: Mapping the OOP Pillars

• Our design leverages the four pillars of Object-Oriented Programming to ensure clean separation of concerns: Abstraction: The QueueBroker abstracts partition management, offsets, and thread coordination.
• Producers simply call publish(topic, message) without managing lock states.
• The details of how messages are stored, indexed, and synchronized are completely hidden from the publisher.
• Encapsulation: The message list and read offsets are encapsulated inside TopicPartition and ConsumerGroup respectively.

Observable outcome: Encapsulation: The message list and read offsets are encapsulated inside TopicPartition and ConsumerGroup respectively.

Stage 4 — Deep Dive: Thread Synchronization and Consumer Offsets

• Let's look at the concurrency mechanics required to handle high-throughput message ingestion.
• The Internals of Lock-Free Offsets and Blocking Queues To guarantee thread-safety without creating bottlenecks, we isolate lock boundaries.
• If we locked the entire QueueBroker or Topic during a write, we would serialise all operations.
• Producers writing to unrelated topics would block each other.

Observable outcome: Producers writing to unrelated topics would block each other.

Stage 5 — Visualizing Concurrency: The Producer-Consumer Flow

• The flowchart below visualizes the thread interaction between producers appending messages and consumers waiting for new data.
• This concurrency sequence diagram illustrates how producers and consumers synchronize.
• A consumer thread attempting to read from an empty partition enters a waiting state by calling await() on the partition's condition variable.
• When a producer thread locks the partition and appends a message, it invokes signal() .

flowchart TD
    Producer[Producer Thread] -->|lock| Partition[Topic Partition]
    Partition -->|append message| MsgList[Message List]
    MsgList -->|signal hasMessages| Consumer[Consumer Thread]
    Consumer -->|await on hasMessages| Idle[Waiting State]
    MsgList -->|read message| Consumer

Observable outcome: When a producer thread locks the partition and appends a message, it invokes signal() .

Stage 6 — Decision and Extension: Custom Partition Routing

• In production systems, routing messages to partitions depends on the use case.
• By introducing a routing abstraction, we can dynamically change routing behavior without modifying the broker class.
• We compare standard strategies below: Using a polymorphic RoutingStrategy allows the client to select the correct balancing behavior dynamically based on system demands.

Observable outcome: We compare standard strategies below: Using a polymorphic RoutingStrategy allows the client to select the correct balancing behavior dynamically based on system demands.

End-to-end outcome

After working through the stages, you should be able to explain:

• Keep these best practices in mind when designing multithreaded queues: 1.
• Avoid Global Locks: Place locks inside individual partitions rather than at the topic level to isolate lock contention.
• Partition Isolation: Place locks inside individual partitions to support concurrent parallel writes.
• Reentrant Locks: Use Java's ReentrantLock and Condition to block and wake up consumer threads safely.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

Trade-offs and Failure Modes: In-Memory Storage Limits

• Detection: Operating an event queue entirely in-memory introduces critical trade-offs: JVM Memory Pressure (Out of Memory): Since messages are stored inside Java lists, a slow consumer group will cause partitio…
• User or operator impact: Operating an event queue entirely in-memory introduces critical trade-offs: JVM Memory Pressure (Out of Memory): Since messages are stored inside Java lists, a slow consumer group will cause partitio…