Design a Distributed Task Scheduler

A task scheduler looks trivial until you say the numbers out loud. "Run this job every minute" is easy for one machine and one job. "Fire one million jobs across the next 24 hours, within 200ms of their scheduled time, surviving a node crash, never silently dropping a firing" is a distributed-systems problem wearing a cron costume.

The interview reward comes from treating the scheduler as a chain of explicit promises: a job scheduled for 09:00 fires at 09:00, exactly one node enqueues it, a crashed leader does not lose the firing, and a slow handler does not pile up behind itself. Every design decision below exists to keep one of those promises.

What the scheduler actually has to do#

Strip away the buzzwords and a scheduler owns four verbs: register a job (one-shot at a timestamp, or recurring via cron), find the jobs due right now, dispatch them to workers, and record that they ran. The API is small:

The non-functional promises are where the design lives:

Name these first. Choosing Cassandra or Kafka before you have written down "fire within 200ms" is how interviews go sideways.

Storing a cron schedule without getting burned by time#

Consider a real request: run job X every day at 09:00 in Europe/Berlin. The naive move is to compute the next UTC firing once and store 2026-07-17T07:00:00Z. That is a bug waiting for the last Sunday in October.

Store the cron expression plus the IANA timezone string (Europe/Berlin), never a fixed UTC offset. Berlin is UTC+1 in winter and UTC+2 in summer, so a stored offset is wrong for half the year. Resolve the next firing to UTC lazily, on each scheduler tick, using the current tzdata version — that way daylight-saving transitions are handled by your cron library instead of by a stale precomputed timestamp.

Two edge cases need an explicit, documented policy:

• Spring-forward skips 02:30 entirely. A job scheduled for 02:30 that day either skips or fires at 03:00 — you must pick.
• Fall-back runs 02:30 twice. The job fires once, twice, or you dedupe — again, your call, but it must be a call, not an accident.

That last_fired_at, persisted durably, is the single most important column in the whole design — it is what makes crash recovery possible.

The data path for a million jobs#

A SELECT * FROM jobs WHERE next_fire_at <= now() every 200ms across a million-row table is a full scan that melts your database. The fix is to shard jobs by time bucket so each poll touches only the jobs due in a narrow window.

Use a store like Cassandra or DynamoDB with partition_key = floor(fire_ts / 60s) and cluster_key = fire_ts. Each minute-bucket partition holds only the jobs due in that minute — a few hundred rows, not a million. A dispatcher polls the current and next bucket roughly every 200ms, pulls the due rows, and pushes them onto a durable worker queue (Kafka or SQS). One range scan per tick, bounded in size.

For sub-second precision on jobs due imminently, layer hierarchical timing wheels in memory in front of the durable store. A timing wheel is an array of buckets you rotate through like a clock hand — inserting and firing a timer is O(1), no per-tick sort. Jobs firing within the next few minutes live in the wheel; everything farther out spills to the bucketed store and gets pulled into the wheel as its time approaches. Hot path O(1), cold path a single range scan.

Who fires the job? Leader election#

Run N scheduler nodes for availability and you create a new hazard: two nodes both notice the 09:00 firing and enqueue it twice. You want redundancy and a single writer per firing.

Elect a leader with a coordination service — ZooKeeper, etcd, or Consul — using a lease-based election. Nodes race to create an ephemeral key; the winner holds a time-bounded lease and renews it; followers watch for the lease to expire and take over instantly when it does. The subtle failure is a stale leader: a node whose lease expired during a GC pause but who is still alive and still trying to enqueue. Guard against it with a fencing token — a monotonically increasing lease epoch attached to every enqueue, so the downstream queue rejects writes carrying an old epoch.

A SET NX EX lock in Redis is the tempting shortcut. It works for low stakes, but it has no fencing and can split-brain under a network partition, letting two "leaders" both act. Mention it as the pragmatic option and name its weakness — that contrast is exactly what interviewers want.

Surviving a crash: misfire recovery#

The leader dies at 08:59:30. A follower notices the lease expired and takes over at 09:01:00. The 09:00 firings happened during the gap — who fires them?

This is why last_fired_at is persisted. On startup the new leader scans for every job where next_fire_at <= now AND last_fired_at < next_fire_at — the jobs that were due but never marked as fired — and enqueues the misses. What it does with them is a per-job misfire policy:

• fire-once-catchup — run a single catch-up firing (most recurring jobs).
• fire-all-missed — replay every skipped firing (backfill/ETL jobs that must not skip a window).
• skip — drop stale firings entirely (real-time jobs where a two-minute-old trigger is worthless).

Bound the catch-up window. If a node is down for a week, fire-all-missed on a per-minute job would unleash ten thousand stale executions in one thundering burst. Cap it.

Delivery guarantees: at-least-once and why exactly-once is a trap#

Queues, worker crashes, and lost acks all mean a task can be delivered more than once. Real schedulers — Sidekiq, Celery, AWS EventBridge — all ship at-least-once, and they push the burden onto handlers: make them idempotent.

True exactly-once across independent processes needs a two-phase commit or a transactional outbox spanning both the queue and the side effect, which only works cleanly inside a single storage system and costs latency and complexity everywhere else. The production answer is at-least-once plus an idempotency key stored alongside the side effect. Derive the key from job_id + fire_ts; before performing the effect, check whether that key already committed. Duplicates get absorbed instead of double-charging a customer.

Two deep dives interviewers love#

Preventing pileup. A job runs every minute but its handler occasionally takes 10 minutes. Without a guard you get overlapping copies stampeding a downstream service. Enforce a per-job concurrency cap — usually 1 for recurring jobs — via a distributed lock keyed on job_id that a worker must hold before running. Overlapping firings then skip, coalesce into a single pending firing, or queue to a bounded depth. Emit a backlog = scheduled_firings - completed_firings metric and alert when it climbs; that is your early warning before the handler recovers and floods everything at once.

Cancellation of an in-flight task. Model each firing as a state machine — SCHEDULED → CLAIMED → RUNNING → DONE — with a cancel_requested flag flipped by a single compare-and-swap. If the task is still SCHEDULED, cancellation is instant: CAS it to CANCELLED and the dispatcher skips it. If it is already RUNNING, you cannot yank it out from under the worker; set cancel_requested and let the worker poll it cooperatively at checkpoints, with a hard deadline enforced by revoking the worker's lease if it ignores the signal. Return the caller a cancel-accepted versus cancel-completed distinction so they know whether to wait for a terminal state.

Fairness: priority without starvation#

If tasks carry priority, strict priority queues are a trap — the moment high-priority traffic exceeds worker capacity, low-priority tasks starve forever. Use weighted fair queuing instead: keep separate high/med/low queues and have workers dequeue on a weighted round-robin like 6:3:1, so low-priority work always gets a slice. The elegant single-queue variant assigns each task a virtual deadline of enqueue_time + base_latency / priority_weight, so an old low-priority task eventually out-ranks a fresh high-priority one. Both bound the worst case; strict priority does not.

How this shows up in interviews#

Task scheduler questions almost always start narrow — "build cron as a service" — and then the interviewer pulls one thread until it snaps. The reliable follow-ups: how do you avoid double-firing across nodes (leader election + fencing), what happens when the scheduler crashes at 08:59 (persisted last_fired_at + misfire policy), the handler is slower than its interval (concurrency cap), and how do you hit sub-second timing at a million jobs (bucketed partitions + timing wheels).

Walk one firing end-to-end: registration writes the cron + tz, the dispatcher resolves the next UTC time each tick, the leader enqueues under a fencing token, a worker claims it under a per-job lock, runs it idempotently, and stamps last_fired_at. Name the source of truth — the durable job store — and state the trade-off you would reverse if the workload changed, such as dropping timing wheels if 1-second granularity were acceptable.

Further learning#

• Distributed Task Scheduler System Design — codeKarle's whiteboard walkthrough of the same architecture, a good visual companion to this guide.
• Practice the interactive version: practice this topic on YeetCode.
• Related core designs: Design a Leaderboard, Design a URL Shortener, Design a Rate Limiter, and Design a Key-Value Store.

FAQ#

Why is exactly-once delivery impractical for a task scheduler?#

Achieving true exactly-once across a queue and a separate side effect requires a two-phase commit or a transactional outbox that spans both systems, which only works cleanly when everything lives in one storage engine and adds latency and coordination cost everywhere else. Production schedulers like Sidekiq, Celery, and AWS EventBridge instead guarantee at-least-once and require idempotent handlers. Storing an idempotency key derived from job_id + fire_ts alongside the side effect lets you absorb duplicate deliveries without double-executing the work, which is cheaper and more robust than chasing exactly-once semantics.

How do you stop two scheduler nodes from firing the same job twice?#

Elect a single leader with a lease-based election backed by ZooKeeper, etcd, or Consul: nodes race for an ephemeral key, the winner renews a time-bounded lease, and followers take over the instant it expires. The danger is a stale leader whose lease lapsed during a pause but who keeps trying to enqueue, so attach a fencing token — a monotonically increasing lease epoch — to every enqueue and have the downstream queue reject writes carrying an old epoch. A Redis SET NX EX lock is a simpler substitute but lacks fencing and can split-brain under a partition.

How does a scheduler recover firings missed while a node was down?#

Persist each job's last_fired_at in durable storage rather than memory. When a new leader starts up it scans for jobs where next_fire_at <= now but last_fired_at < next_fire_at, meaning they were due but never marked fired, and re-enqueues them according to a per-job misfire policy. That policy is fire-once-catchup for typical recurring jobs, fire-all-missed for backfill jobs that cannot skip a window, or skip for real-time jobs where a stale trigger is useless. Always bound the catch-up window so a long outage does not release thousands of stale firings in one burst.

How does the design hit sub-second timing for a million scheduled jobs?#

Shard jobs into time buckets with partition_key = floor(fire_ts / 60s) in a store like Cassandra or DynamoDB, so each minute partition holds only that minute's jobs and a dispatcher range-scans one small partition per tick instead of the whole table. Poll the current and next bucket roughly every 200ms and push due rows onto a durable worker queue. For jobs firing imminently, layer in-memory hierarchical timing wheels in front of the store: insertion and firing are O(1) with no per-tick sort, and jobs spill to the durable buckets when their firing time is far out.

How do you prevent a slow recurring job from piling up?#

Enforce a per-job concurrency cap — typically 1 for recurring jobs — using a distributed lock keyed on job_id that a worker must acquire before running. When a new firing arrives while the previous one still holds the lock, it either skips, coalesces into a single pending firing, or queues up to a bounded depth, so you never accumulate overlapping copies. Track a backlog metric of scheduled minus completed firings and alert when it grows; that signals a handler running slower than its interval before the queue floods downstream services when the handler finally recovers.