High-Level Design: Building a Real-Time Ad Click Aggregator at Scale

Scenario context

Scaling an ad click aggregator requires processing massive event streams (billions of clicks per day) with exactly-once delivery guarantees. We achieve this using Kafka-based event ingestion, Apache Flink stateful stream aggregation, and timeseries databases.

Observable outcome: API Design We define the endpoints for click collection and dashboard retrieval: Database Schema (Timeseries Store) We store aggregated metrics in a timeseries-optimized database like ClickHouse or InfluxDB: Cache Schem…

Stage 3 — Architectural Blueprint: High-Level System Flow

• The diagram below maps the components of our ad click aggregator: This system diagram illustrates the architecture of our ad click aggregation platform.
• Raw click events from browsers are received by a click collection service behind a load balancer.
• The service runs a quick deduplication check in Redis and immediately appends the event to an Apache Kafka broker.
• Apache Flink consumes from Kafka, maintaining stateful sliding window aggregations in-memory before flushing the final aggregates to a ClickHouse timeseries database.

graph TD
    Client[User Browser] -->|Click Event| LB[Load Balancer]
    LB -->|Forward| API[Click Collection Service]
    API -->|Write Dedup Key| Redis[Deduplication Redis]
    API -->|Buffer Event| Kafka[Apache Kafka Cluster]
    Kafka -->|Stream Consumption| Flink[Apache Flink Engine]
    Flink -->|Write Aggregations| TSDB[ClickHouse Timeseries DB]
    Dashboard[Query Dashboard] -->|Get Metrics| QuerySvc[Query Service]
    QuerySvc -->|Read Metrics| TSDB

Observable outcome: Apache Flink consumes from Kafka, maintaining stateful sliding window aggregations in-memory before flushing the final aggregates to a ClickHouse timeseries database.

Stage 4 — Deep Dive: Exactly-Once Processing and Streaming Windows

• Guaranteeing exact metrics under flash load requires coordinate states across the processing pipeline.
• The Internals of Sliding Windows and Deduplication To count clicks exactly once, the system uses two layers of defense: 1.
• Edge Deduplication: When a click arrives, the Collection Service builds a fingerprint hash using (user id + ad id + timestamp minute) .
• It checks if this key exists in Redis using an atomic transaction.

Observable outcome: It checks if this key exists in Redis using an atomic transaction.

Stage 5 — Write and Read Path Sequences

• Write Path Flow (Click Ingestion & Processing) 1.
• The client clicks an ad, triggering a POST /clicks request.
• The Collection Service validates the click token and checks the Redis deduplication cache.
• The Collection Service publishes the event to Apache Kafka using ad id as the partition key.

Observable outcome: The Collection Service publishes the event to Apache Kafka using ad id as the partition key.

Stage 6 — Real-World Implementation: Ad Click Processing at Scale

• Real-world advertising giants like Google and Meta split processing into hot and cold paths: The Hot Path (Real-time): Stream aggregators like Flink compute click metrics over short sliding windows to update dashboards…
• The Cold Path (Batch): Raw click events are saved to Hadoop or Snowflake.
• Nightly batch jobs reconcile the day's traffic to calculate final advertiser billing.

Observable outcome: Nightly batch jobs reconcile the day's traffic to calculate final advertiser billing.

End-to-end outcome

After working through the stages, you should be able to explain:

• Edge Deduplication: Filter click spam early using Redis atomic fingerprint lookups.
• Stateful Streaming: Use Apache Flink to aggregate metrics in-memory before writing to the database.

Alternatives considered

Decision Guide: Lambda vs. Kappa Architecture

Use this decision table to select the correct streaming architecture for your application.

Failure and degraded modes

Trade-offs and Failure Modes: Latency vs. Accuracy in Streaming

• Detection: Choosing a streaming architecture requires balancing latency, accuracy, and operational overhead: Our design uses Flink stateful processing to achieve real-time metrics with exactly-once guarantees.
• User or operator impact: Choosing a streaming architecture requires balancing latency, accuracy, and operational overhead: Our design uses Flink stateful processing to achieve real-time metrics with exactly-once guarantees.

Lessons Learned: Production Streaming Pitfalls

• Detection: Avoid these standard mistakes when deploying stream aggregators: Ignoring Late-Arriving Data: Network delays can cause click events to arrive minutes late.
• User or operator impact: Avoid these standard mistakes when deploying stream aggregators: Ignoring Late-Arriving Data: Network delays can cause click events to arrive minutes late.