System Design HLD Example: Real-Time Leaderboard

Scenario context

Real-time leaderboards for 10M+ active users require an in-memory ranking engine. Redis Sorted Sets (ZSET) are the industry standard, providing $O(\log N)$ updates and rank lookups via an internal Skip List data structure. Relational databases fail at this scale because calculating rank is an $O(N)$ operation (counting rows with higher scores) that becomes a bottleneck during peak traffic spikes.

This walkthrough focuses on **Deep Dive: Why Redis Sorted Sets Achieve What SQL Cannot**.

Before making a design choice, clarify:

- The performance gulf between Redis ZSET and SQL for leaderboard rank queries is not a matter of hardware speed — it is a fundamental algorithmic difference.
- Understanding the internals of the Redis Skip List reveals why this gap exists and why no amount of database tuning closes it.
- Internals: The Skip List and Hash Map That Power ZREVRANK A Redis Sorted Set is implemented internally as two data structures working in tandem : 1.

**Concepts in play:** caching, system-design-hld-architectures, transaction-isolation-distributed-acid, database-indexing

## Stage 1 — The Last-Minute Snipe

- Imagine it’s the final minute of a week-long gaming tournament.
- Ten million players are grinding for the top spot.
- You’re currently ranked 10, and you just scored 5,000 points.
- You expect to see your rank jump to 2 immediately.

*Applies [**caching**](/concept/caching) to the decision above.*

**Observable outcome:** You expect to see your rank jump to 2 immediately.

## Stage 2 — Real-Time Leaderboard: Use Cases & Requirements

- Actors & Journeys - Player: Plays matches, earns points, and checks their global and friends-list ranking.
- - Game Server: The authoritative source that validates gameplay and submits the final score to the leaderboard service.
- - Tournament Admin: Configures seasons, resets scores, and monitors for cheating behavior.
- Functional Requirements - Score Submission: Update a player's score after a match.

*Applies [**system-design-hld-architectures**](/concept/system-design-hld-architectures) to the decision above.*

**Observable outcome:** Functional Requirements - Score Submission: Update a player's score after a match.

## Stage 3 — Basics: How Ranking Differs from Sorting

- At first glance, a leaderboard seems like a simple sorting problem.
- Just sort the table by score and pick the top rows.
- However, at scale, there is a massive difference between sorting and ranking .
- Sorting: Ordering a set of elements.

*Applies [**transaction-isolation-distributed-acid**](/concept/transaction-isolation-distributed-acid) to the decision above.*

**Observable outcome:** Sorting: Ordering a set of elements.

## Stage 4 — Mechanics: The Logic of Score Updates

- The core mechanic of a modern leaderboard is the Accumulation and Fan-out of score events.
- - Incremental Updates: Instead of overwriting the score, we often send "score deltas" (e.g., +50 points).
- This allows for easier concurrent updates.
- - Tie-Breaking: To ensure deterministic ranking, we use a composite score.

*Applies [**database-indexing**](/concept/database-indexing) to the decision above.*

**Observable outcome:** - Tie-Breaking: To ensure deterministic ranking, we use a composite score.

## Stage 5 — Estimations & Design Goals

- The Math of Massive Ranking - Total Players: 100 Million.
- - Daily Active Users (DAU): 10 Million.
- - Peak Write Throughput: 50,000 score updates/sec.
- - Peak Read Throughput: 100,000 rank queries/sec.

*Applies [**caching**](/concept/caching) to the decision above.*

**Observable outcome:** - Peak Read Throughput: 100,000 rank queries/sec.

## Stage 6 — High-Level Design: The Ranking Pipeline

- The architecture separates the fast-path ranking engine from the durable-path storage.
- The Ranking Pipeline above separates two concerns that must never share a bottleneck: the write path (score ingestion via Kafka → Score Processor → Redis ZSET) and the read path (Leaderboard API → Redis ZREVRANK).
- The Game Server never writes directly to Redis — it goes through the Ingestion Service and Kafka, ensuring durability before the score is considered committed.
- If Redis is unavailable, the API falls back to Postgres for correctness at the cost of higher latency.

*Applies [**system-design-hld-architectures**](/concept/system-design-hld-architectures) to the decision above.*

```architecture
{
  "version": 1,
  "title": "High-Level Design: The Ranking Pipeline - Subsystem Overview",
  "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.",
  "view": "high-level",
  "pattern": "microservices",
  "direction": "LR",
  "groups": [
    { "id": "edge", "label": "Edge Traffic Ingress", "tone": "slate" },
    { "id": "services", "label": "Stateless Microservices", "tone": "blue" },
    { "id": "data", "label": "Datastore Infrastructure", "tone": "green" }
  ],
  "nodes": [
    { "id": "gw", "component": "api-gateway", "label": "API Gateway", "group": "edge", "badge": "TLS & Auth" },
    { "id": "app", "component": "workflow-service", "label": "Domain Service", "group": "services", "badge": "Stateless" },
    { "id": "cache", "component": "redis", "label": "Redis Cache", "group": "data", "badge": "Sub-ms Reads" },
    { "id": "db", "component": "postgres", "label": "Transactional DB", "group": "data", "badge": "ACID Log" }
  ],
  "edges": [
    { "from": "gw", "to": "app", "kind": "request", "label": "gRPC Request" },
    { "from": "app", "to": "cache", "kind": "read", "label": "Cache Lookups" },
    { "from": "app", "to": "db", "kind": "write", "label": "Durable Write" }
  ],
  "takeaway": {
    "title": "Architecture Boundary Invariant",
    "body": "Stateless compute nodes scale horizontally while shielding underlying database clusters behind fast cache layers."
  }
}

graph TD
    Game[Game Server] -->|Score| Ingest[Ingestion Service]
    Ingest -->|Kafka| MQ[Message Queue]
    MQ --> Processor[Score Processor]

    Processor -->|ZADD| Redis[(Redis ZSET Index)]
    Processor -->|Write| Postgres[(Postgres: Score History)]

    Player -->|Get Rank| API[Leaderboard API]
    API -->|ZREVRANK| Redis
    API -.->|Fallback| Postgres

Observable outcome: If Redis is unavailable, the API falls back to Postgres for correctness at the cost of higher latency.

End-to-end outcome

After working through the stages, you should be able to explain:

• Lesson 1 — The composite score encoding must be documented prominently.
• A developer joining the team months later will see a score like 892000001707868800 in Redis and have no idea it encodes both points and a timestamp.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

In-Memory Speed vs. Durability: Trade-offs in Leaderboard Design

• Detection: Critical Failure Mode — The Redis Restart Without Persistence: If the Redis instance restarts and AOF (Append Only File) persistence was not enabled, the entire ZSET is lost.
• User or operator impact: Critical Failure Mode — The Redis Restart Without Persistence: If the Redis instance restarts and AOF (Append Only File) persistence was not enabled, the entire ZSET is lost.