Design a Real-Time
Log Analytics Pipeline.
SAY THIS: “Before proposing a topology for Design a Real-Time Log Analytics Pipeline, I’ll clarify the core functional requirements, burst characteristics, data volume, and availability SLOs to define rigid system boundaries.”
What the system does
- Ingest 600K events/sec sustained and 2.5M events/sec burst with zero silent data loss for error/audit logs and bounded disk buffering on nodes.
- Make ingested logs queryable for aggregations and trace_id/keyword search within <= 5 seconds end-to-end.
- Support 7 days of hot SSD analytics (< 1.5s p95 for 24-hour range queries) and 365 days of queryable cold archive in object storage.
- Protect the cluster from noisy-neighbor tenants and schema explosion (unbounded dynamic JSON keys).
What to agree on
- Raw log volume is 50 TB/day; hot SSD budget is capped at 80 TB total across replicas.
- Node-local log agents (Fluent Bit) must stay under 150 MB RAM and 2% CPU per node.
- Network egress between availability zones must be minimized via batching and compression.
SAY THIS: “Let’s establish order-of-magnitude sizing for peak throughput, memory footprint, and network ingress to size the critical latency path before making technology choices.”
At an illustrative 200 bytes per record/session, high-volume state requires dedicated primary in-memory partitions. Add replicas, allocator overhead, policy indexes, and headroom.
Always validate throughput and p99 latency targets under skewed traffic distributions and replica failovers.
Keep state shared and partitioned.
SAY THIS: “I’ll separate low-latency ephemeral cache state from authoritative durable storage, ensuring strict consistency where needed and partitioning for scale.”
Cache / Ephemeral state
state:{partitionKey}:activeWindow
- In-memory key-value cache (Redis Cluster / Memcached).
- TTL expiration matching window validity bounds.
- Hash tags to co-locate keys for atomic Lua scripts or transactions.
Durable / Authoritative store
- Distributed NoSQL / relational database (DynamoDB, PostgreSQL).
- Durable write-ahead log with replica failover.
- Cache-aside pattern: read misses warm cache from durable store.
- Asynchronous CDC / event streaming to telemetry sinks.
Local-only node state is not a shared cluster quota: N nodes could admit roughly N × the configured limits. Ensure state co-location and transactional boundaries align with partition keys.
SAY THIS: “The gateway tier authenticates incoming traffic, validates payloads, and routes requests to decoupled services backed by partitioned caching and authoritative storage tiers.”
Use tight timeouts, circuit breakers, and rate-limiting middleware at ingress to protect downstream services from cascading collapse.
Define the contract.
SAY THIS: “The API contract establishes idempotent mutation semantics, clear error taxonomy, and explicit status codes so upstream clients can safely retry.”
Success Path
Forward the request or return HTTP 200/201. Include standard tracking and rate/quota telemetry headers in the response.
Failure & Fallback
Use explicit HTTP status codes: 400 for invalid input, 429 for quota violations, and 503 with Retry-After during maintenance or partitions.
View example API payload contract
HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: max-age=300
{
"status": "success",
"data": { "id": "res_98a7f", "processedAt": "2026-11-11T12:00:00Z" }
}Include idempotency keys on write endpoints to guarantee at-most-once processing across network retries.
SAY THIS: “For the critical execution path, we enforce idempotency and concurrency controls using atomic primitives, avoiding lock contention and split-brain states.”
- Put low-cardinality filter columns first in ORDER BY to maximize prefix compression and granule skipping.
- Add bloom_filter on trace_id and tokenbf_v1 on message for fast point lookups.
- Configure TTL ts + INTERVAL 7 DAY TO DISK 's3_cold' for seamless hot-to-cold migration.
| STRATEGY / PATTERN | WHY CHOOSE IT? | TRADE-OFF / COST |
|---|---|---|
| Pattern 1 | Dedicated Kafka topic per enterprise tenant (increases ZooKeeper/KRaft partition metadata overhead). | Coordinate partition dependencies |
View reference production code implementation
CREATE TABLE logs_distributed (
ts DateTime64(3, 'UTC') CODEC(Delta, ZSTD(1)),
tenant_id LowCardinality(String),
service LowCardinality(String),
level Enum8('DEBUG'=1, 'INFO'=2, 'WARN'=3, 'ERROR'=4, 'FATAL'=5),
trace_id String CODEC(ZSTD(1)),
message String CODEC(ZSTD(3)),
attributes Map(String, String) CODEC(ZSTD(3)),
INDEX idx_trace trace_id TYPE bloom_filter(0.01) GRANULARITY 4,
INDEX idx_msg message TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 4
) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/logs', '{replica}')
PARTITION BY toYYYYMMDD(ts)
ORDER BY (tenant_id, service, level, ts)
TTL toDateTime(ts) + INTERVAL 7 DAY TO VOLUME 's3_cold',
toDateTime(ts) + INTERVAL 365 DAY DELETE
SETTINGS index_granularity = 8192;Avoid client-side read-modify-write loops without concurrency control. Use atomic scripts, row-level locks, or conditional mutations at the storage tier to guarantee linearizability.
SAY THIS: “The biggest operational risks involve node failover, partition skew, network isolation, and cascading downstream backpressure. Here is how we mitigate each.”
Salting spreads load evenly across brokers, at the cost of losing strict per-tenant global ordering (acceptable since logs carry millisecond timestamps).
Unbounded Map keys or 10 MB stack-trace strings exhausting Flink/ClickHouse insert memory.
Close with the trade-offs.
SAY THIS: “In closing, we optimized for sub-millisecond p99 latency and high availability under partition. We trade strict distributed linearizability for predictable throughput and operational simplicity.”
If the interviewer asks…
- “Need stronger consistency?” Introduce synchronous consensus or linearizable leases; accept higher tail latency.
- “Need lower cross-region latency?” Route reads to local replicas with asynchronous replication; accept bounded staleness.
Keep the conclusion focused
- What guarantees are strictly enforced at runtime?
- What happens during network partitions or node failures?
- Which CAP trade-off was intentionally chosen and why?
Design a multi-tenant log ingestion and analytics platform sustaining 600K events/sec (50 TB/day) with sub-3-second search latency for hot 7-day logs and 365-day S3 cold retention.
“Decouple bursty edge shippers from columnar storage with a zstd-compressed Kafka buffer, enforce strict cardinality limits on dynamic JSON keys, and pair ClickHouse MergeTree + bloom filters with S3 Parquet tiering.”
ACID Transaction Fundamentals: fundamentals
When is ACID Transaction Fundamentals the right default for a production team?