Design a distributed log ingestion and real-time tailing pipeline capable of ingesting 5 million events/sec (5 GB/s sustained, 15 GB/s burst) from thousands of microservice producers, persisting durable partitioned commit logs, serving real-time consumer groups with sub-50 ms p99 tail latency, and tiering cold segments to object storage without exhausting broker CPU or dropping acknowledged events.
From patterns to decisions.
Combine producer micro-batching, token-bucket backpressure, sequential page-cache appends, and zero-copy `sendfile()` consumer tailers with offset checkpointing and object-storage cold tiering.
Throughput, Fan-Out, and Back-of-the-Envelope Sizing.
- Ingest: 5M events/s × 1 KB avg = 5 GB/s sustained (15 GB/s burst, 3× headroom).
- Replication (RF=3): Leader receives 5 GB/s and replicates 2× to followers = 10 GB/s inter-broker traffic.
- Consumer Fan-Out (3×): 15 GB/s outbound read bandwidth.
- Why standard B-Tree DBs fail: Random 4 KB page updates and WAL + index amplification collapse under 5M inserts/s; only sequential append-only commit logs hit NVMe/NIC line rate.
Producer Micro-Batching & Token-Bucket Admission Control.
Instead of sending 5M individual RPCs/sec, producer SDKs group records per partition (batch.size = 64 KB, linger.ms = 10) and compress batches with zstd (typically 3–4× reduction, cutting wire ingress from 5 GB/s to ~1.5 GB/s).
At the Ingest Gateway, each tenant has a token-bucket rate limiter backed by per-partition queue-depth signals:
- If broker page-cache dirty ratio or replication lag exceeds threshold, the gateway returns
429 Too Many RequestswithRetry-After. - Producers apply bounded exponential backoff with full jitter and spill locally to a bounded ring buffer before dropping/sampling low-priority debug logs.
Segmented Append-Only Commit Log & Sparse Offset Index.
Each topic is sharded into partitions; each partition is an ordered sequence of immutable 1 GB .log segments plus one active tail segment:
.logfile: Raw length-prefixed record batches appended sequentially..indexfile: Sparse memory-mapped index mapping every ~4 KB of log bytes (offset -> physical_byte_position). Finding any offset requires a binary search in RAM followed by a single sequential read.- Durability: Synchronous quorum ack (
acks=all, min-ISR=2) relies on multi-AZ replica memory + periodic OS page-cache flush, avoiding per-batchfsync()stalls unless strict financial durability is required.
Partitioned Commit Log Segment Append & Sparse Index
Zero-Copy sendfile() Consumer Tailing & Page-Cache Isolation.
Because real-time consumers tail within seconds of the producer head, the active .log segment is already hot in the Linux Kernel Page Cache.
- Brokers serve consumer
Fetch(partition, offset, max_bytes)requests viasendfile(), streaming bytes straight from the page cache to the consumer TCP socket via NIC scatter-gather DMA. - Zero user-space allocation: The broker JVM/Go/Rust process never allocates heap buffers for fetched payloads, eliminating GC pauses under 15 GB/s fan-out.
- Cold-tail protection: Backfill consumers reading hours-old data are tagged to bypass or
POSIX_FADV_DONTNEEDpages—or routed to S3-backed tier readers—so they never evict the hot tail.
Idempotent Batch Processing with Bounded Retry, Backoff & DLQ.
At-least-once tailing requires committing offsets only after the downstream sink succeeds (never auto-commit before processing). Poison-pill records must be bounded and diverted to a DLQ rather than blocking the partition forever.
import time, random
def process_batch_with_dlq(records, sink, dlq, checkpoint_store, partition, max_retries=3):
for rec in records:
delivered = False
for attempt in range(max_retries):
try:
sink.write_idempotent(rec.idempotency_key, rec.payload)
delivered = True
break
except TransientSinkError:
backoff = min(2.0, (0.05 * (2 ** attempt)) + random.uniform(0, 0.05))
time.sleep(backoff)
except SchemaValidationError as err:
dlq.publish(partition, rec.offset, rec.payload, reason=str(err))
delivered = True # Diverted to DLQ; safe to advance offset
break
if not delivered:
dlq.publish(partition, rec.offset, rec.payload, reason='max_retries_exhausted')
checkpoint_store.commit_offset(partition, rec.offset + 1)Immutable Segment Tiering to Object Storage (S3).
Keeping 30–90 days of 5 GB/s logs on replicated NVMe drives would require petabytes of expensive local SSDs. Instead:
- Once a 1 GB segment rolls over and is immutable, an async uploader streams
(segment.log, segment.index)to Object Storage (S3) and records the remote manifest in metadata storage. - Local NVMe retains only the last 4–24 hours of hot segments.
- Rebalances and broker replacements become fast because brokers only re-replicate the hot tail, while historical replays fetch byte ranges directly from S3.
Hot Partitions, Slow Consumers, and ISR Shrinking.
- Hot Partition Skew: If a single
tenant_idgenerates 40% of traffic, hash partitioning pins one broker. Mitigate with salted keys (tenant_id#rand(0..k)) for stateless logs or adaptive shard splitting. - Follower Lag & ISR Shrink: A degraded disk on one replica slows
acks=allwrites. Enforcereplica.lag.time.max.msto drop slow followers from the ISR cleanly while alerting on reduced redundancy. - Rebalance Storms: Long consumer GC pauses trigger group rebalances. Use incremental cooperative rebalancing and static group membership (
group.instance.id).
Practice: Make the trade-off yourself.
A backfill job starts reading 12-hour-old log segments from disk, and real-time consumer p99 tail latency immediately jumps from 15 ms to 450 ms. What is the most likely root cause?