System Design HLD Example: Distributed Job Scheduler

Scenario context

A distributed job scheduler ensures tasks fire reliably using a durable Job Store with a next_fire_time index. To handle multiple scheduler instances without double-firing, we use optimistic row-level locking (UPDATE WHERE status='SCHEDULED'). By decoupling the Trigger Engine from the Worker Pool via a message queue (Kafka/SQS), the system achieves independent scaling and guaranteed recovery from…

This walkthrough focuses on The "Ghost Task" Problem.

Before making a design choice, clarify:

• Imagine you’re building a billing system for a global SaaS platform.
• Every month, on the 1st at midnight, your system must trigger 10 million subscription renewals.
• If your scheduler is a simple in-memory cron job (like a basic Timer in Java or a cron tab on a single server), and the server restarts at 11:59 PM, all 10 million jobs vanish.

Concepts in play: system-design-hld-architectures, caching, database-indexing, message-queues

Stage 1 — Distributed Job Scheduler: Use Cases & Requirements

• Actors & Journeys - Client Application: Submits tasks to be executed later (e.g., "Send this email in 2 hours").
• • Trigger Engine: The "clock" of the system that identifies which tasks are due for execution.
• • Worker Pool: The fleet of stateless execution nodes that perform the actual work (e.g., calling an external API).
• • Admin Console: Provides visibility into the job pipeline, allowing for manual retries and status monitoring.

Observable outcome: - Admin Console: Provides visibility into the job pipeline, allowing for manual retries and status monitoring.

Stage 2 — Basics: How Time-Based Triggers Work

• At its simplest level, a scheduler is a Priority Queue where the "Priority" is the next fire time .
• The Polling Loop: A background thread wakes up every second, queries for all jobs where next fire time <= now , and marks them for execution.
• The Precision Problem: If your polling loop takes 2 seconds but you poll every 1 second, the polls start to overlap.
• If your database query takes 5 seconds, your "real-time" scheduler is now lagging significantly.

Observable outcome: If your database query takes 5 seconds, your "real-time" scheduler is now lagging significantly.

Stage 3 — Mechanics: The Lifecycle of a Distributed Task

• The life of a scheduled job follows a precise state machine to ensure reliability: 1.
• SCHEDULED: The job is persisted in the DB and indexed in Redis.
• ACQUIRED/RUNNING: A scheduler instance has claimed the job and is currently pushing it to the message queue.
• DISPATCHED: The job is on the queue, waiting for a worker.

flowchart TD
    A["Submitted"] --> B["Scheduled"]
    B --> C["Acquired"]
    C --> D["Dispatched"]
    D --> E["Running"]
    E --> F{"Outcome"}
    F -->|Success| G["Completed"]
    F -->|Error| H{"Retry limit?"}
    H -->|Under limit| I["Retrying"]
    H -->|Exceeded| J["Dead Letter"]
    I --> B
    B --> K["Paused"]
    K --> B

Observable outcome: DISPATCHED: The job is on the queue, waiting for a worker.

Stage 4 — Estimations & Design Goals

• The Math of "Midnight Bursts" - Daily Volume: 100 Million jobs/day.
• • The Hot Minute: 20% of all daily jobs cluster at 00:00 (Midnight) for daily reports and billing.
• • Burst Throughput: 20M jobs / 60 seconds $\approx \mathbf{333,000 \text{ jobs/sec}}$ to be triggered.
• • Storage Requirement: 100M jobs $\times$ 500 bytes (JSON payload + metadata) $\approx \mathbf{50 \text{ GB/day}}$.

Observable outcome: - Storage Requirement: 100M jobs $\times$ 500 bytes (JSON payload + metadata) $\approx \mathbf{50 \text{ GB/day}}$.

Stage 5 — High-Level Design: The Poll-Dispatch-Execute Pattern

• The architecture separates the time-keeping (Trigger) from the work-execution (Worker).
• The diagram above separates the three core roles: the Job API accepts submissions and returns 202 Accepted immediately; the Scheduler Service identifies due jobs and claims them; and the Worker Pool executes them indepe…
• The Redis Sorted Set acts as a lightweight time index for O(log N) polling, while Postgres is the durable source of truth.
• This architecture means each component can fail and recover independently without losing a single job.

graph TD
    Client -->|POST /jobs| API[Job API]
    API -->|Insert| DB[(Job Store: Postgres)]
    API -->|ZADD| Redis[(Redis Trigger Index)]

    subgraph Trigger_Engine
        Sched[Scheduler Service] -->|Poll| Redis
        Sched -->|CAS Lock| DB
        Sched -->|Dispatch| MQ[Message Queue: Kafka]
    end

    MQ --> Workers[Worker Pool]
    Workers -->|Heartbeat| DB
    Workers -->|Update Status| DB

Observable outcome: This architecture means each component can fail and recover independently without losing a single job.

Stage 6 — Deep Dive: How the Scheduler Achieves Exactly-Once Dispatch Without a Global Lock

• The most subtle engineering in a distributed scheduler is the claim mechanism.
• If multiple Scheduler instances are running for high availability, they must race to claim each job without either missing it (never fires) or both claiming it (double fires).
• The solution is Optimistic Locking via Compare-and-Swap (CAS) UPDATE .
• Internals: The Poll-and-Lock Cycle Every second (configurable), each Scheduler instance queries the Redis Sorted Set for all jobs with a score (next fire time as Unix epoch) less than or equal to the current time.

sequenceDiagram
    participant C as Client
    participant API as Job API
    participant DB as Job Store
    participant R as Redis ZSET
    participant S as Scheduler
    participant MQ as Kafka
    participant W as Worker
    C->>API: POST /jobs with payload and schedule
    API->>DB: INSERT job with status=SCHEDULED
    API->>R: ZADD trigger score=next_fire_time
    API-->>C: 202 Accepted
    S->>R: ZRANGEBYSCORE for due jobs
    S->>DB: CAS UPDATE status=ACQUIRED
    S->>MQ: Publish job message to topic
    MQ->>W: Deliver job to consumer
    W->>DB: UPDATE status=SUCCESS

Observable outcome: Internals: The Poll-and-Lock Cycle Every second (configurable), each Scheduler instance queries the Redis Sorted Set for all jobs with a score (next fire time as Unix epoch) less than or equal to the current time.

End-to-end outcome

After working through the stages, you should be able to explain:

• Lesson 1 — Idempotency in Workers is not optional.
• The system provides at-least-once delivery by design — Kafka may redeliver, the Watchdog may re-dispatch a stuck job.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

Reliability vs. Throughput: Trade-offs in Distributed Scheduling

• Detection: The component diagram below shows how the Scheduler Cluster, Coordination Layer, and Worker Pool connect, making clear which tier scales independently under burst load.
• User or operator impact: The component diagram below shows how the Scheduler Cluster, Coordination Layer, and Worker Pool connect, making clear which tier scales independently under burst load.