Skip to main content
← PROBLEMS / SYSTEM-DESIGN
An end-to-end domain learning journey

Design a High-Throughput Distributed Log
Ingestion & Tail Pipeline.

Combine producer micro-batching, token-bucket backpressure, sequential page-cache appends, and zero-copy sendfile() consumer tailers with offset checkpointing and object-storage cold tiering.
Start the journey7 stages
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Design a distributed log ingestion and real-time tailing pipeline capable of ingesting 5 million events/sec (5 GB/s sustained, 15 GB/s burst) from thousands of microservice producers, persisting durable partitioned commit logs, serving real-time consumer groups with sub-50 ms p99 tail latency, and tiering cold segments to object storage without exhausting broker CPU or dropping acknowledged events.

From patterns to decisions.

Combine producer micro-batching, token-bucket backpressure, sequential page-cache appends, and zero-copy `sendfile()` consumer tailers with offset checkpointing and object-storage cold tiering.

Full walkthrough · 7 stages
01
STAGE 01 · REQUIREMENTS & MATHfoundation

Throughput, Fan-Out, and Back-of-the-Envelope Sizing.

  • Ingest: 5M events/s × 1 KB avg = 5 GB/s sustained (15 GB/s burst, 3× headroom).
  • Replication (RF=3): Leader receives 5 GB/s and replicates 2× to followers = 10 GB/s inter-broker traffic.
  • Consumer Fan-Out (3×): 15 GB/s outbound read bandwidth.
  • Why standard B-Tree DBs fail: Random 4 KB page updates and WAL + index amplification collapse under 5M inserts/s; only sequential append-only commit logs hit NVMe/NIC line rate.
02
STAGE 02 · INGEST & BACKPRESSUREfoundation

Producer Micro-Batching & Token-Bucket Admission Control.

Instead of sending 5M individual RPCs/sec, producer SDKs group records per partition (batch.size = 64 KB, linger.ms = 10) and compress batches with zstd (typically 3–4× reduction, cutting wire ingress from 5 GB/s to ~1.5 GB/s).

At the Ingest Gateway, each tenant has a token-bucket rate limiter backed by per-partition queue-depth signals:

  • If broker page-cache dirty ratio or replication lag exceeds threshold, the gateway returns 429 Too Many Requests with Retry-After.
  • Producers apply bounded exponential backoff with full jitter and spill locally to a bounded ring buffer before dropping/sampling low-priority debug logs.
03
STAGE 03 · STORAGE ENGINEintermediate

Segmented Append-Only Commit Log & Sparse Offset Index.

Each topic is sharded into PP partitions; each partition is an ordered sequence of immutable 1 GB .log segments plus one active tail segment:

  • .log file: Raw length-prefixed record batches appended sequentially.
  • .index file: Sparse memory-mapped index mapping every ~4 KB of log bytes (offset -> physical_byte_position). Finding any offset requires a binary search in RAM followed by a single sequential read.
  • Durability: Synchronous quorum ack (acks=all, min-ISR=2) relies on multi-AZ replica memory + periodic OS page-cache flush, avoiding per-batch fsync() stalls unless strict financial durability is required.

Partitioned Commit Log Segment Append & Sparse Index

Compressed BatchActive SegmentSparse .indexClosed 1 GBSegments
04
STAGE 04 · ZERO-COPY READ PATHintermediate

Zero-Copy sendfile() Consumer Tailing & Page-Cache Isolation.

Because real-time consumers tail within seconds of the producer head, the active .log segment is already hot in the Linux Kernel Page Cache.

  • Brokers serve consumer Fetch(partition, offset, max_bytes) requests via sendfile(), streaming bytes straight from the page cache to the consumer TCP socket via NIC scatter-gather DMA.
  • Zero user-space allocation: The broker JVM/Go/Rust process never allocates heap buffers for fetched payloads, eliminating GC pauses under 15 GB/s fan-out.
  • Cold-tail protection: Backfill consumers reading hours-old data are tagged to bypass or POSIX_FADV_DONTNEED pages—or routed to S3-backed tier readers—so they never evict the hot tail.
05
STAGE 05 · SAFE CONSUMER CODEintermediate

Idempotent Batch Processing with Bounded Retry, Backoff & DLQ.

At-least-once tailing requires committing offsets only after the downstream sink succeeds (never auto-commit before processing). Poison-pill records must be bounded and diverted to a DLQ rather than blocking the partition forever.

import time, random

def process_batch_with_dlq(records, sink, dlq, checkpoint_store, partition, max_retries=3):
    for rec in records:
        delivered = False
        for attempt in range(max_retries):
            try:
                sink.write_idempotent(rec.idempotency_key, rec.payload)
                delivered = True
                break
            except TransientSinkError:
                backoff = min(2.0, (0.05 * (2 ** attempt)) + random.uniform(0, 0.05))
                time.sleep(backoff)
            except SchemaValidationError as err:
                dlq.publish(partition, rec.offset, rec.payload, reason=str(err))
                delivered = True  # Diverted to DLQ; safe to advance offset
                break
        if not delivered:
            dlq.publish(partition, rec.offset, rec.payload, reason='max_retries_exhausted')
        checkpoint_store.commit_offset(partition, rec.offset + 1)
06
STAGE 06 · TIERED STORAGEadvanced

Immutable Segment Tiering to Object Storage (S3).

Keeping 30–90 days of 5 GB/s logs on replicated NVMe drives would require petabytes of expensive local SSDs. Instead:

  1. Once a 1 GB segment rolls over and is immutable, an async uploader streams (segment.log, segment.index) to Object Storage (S3) and records the remote manifest in metadata storage.
  2. Local NVMe retains only the last 4–24 hours of hot segments.
  3. Rebalances and broker replacements become fast because brokers only re-replicate the hot tail, while historical replays fetch byte ranges directly from S3.
07
STAGE 07 · FAILURE MODESadvanced

Hot Partitions, Slow Consumers, and ISR Shrinking.

  • Hot Partition Skew: If a single tenant_id generates 40% of traffic, hash partitioning pins one broker. Mitigate with salted keys (tenant_id#rand(0..k)) for stateless logs or adaptive shard splitting.
  • Follower Lag & ISR Shrink: A degraded disk on one replica slows acks=all writes. Enforce replica.lag.time.max.ms to drop slow followers from the ISR cleanly while alerting on reduced redundancy.
  • Rebalance Storms: Long consumer GC pauses trigger group rebalances. Use incremental cooperative rebalancing and static group membership (group.instance.id).
Practice · Explain the consequences

Practice: Make the trade-off yourself.

Question 1 of 3
Score: 0/3

A backfill job starts reading 12-hour-old log segments from disk, and real-time consumer p99 tail latency immediately jumps from 15 ms to 450 ms. What is the most likely root cause?

Recommended Learning Flow