system-design · intermediate

Design a Job Scheduler — Delayed and Recurring Work

Start here

A job scheduler runs work later:

“Send this email in 15 minutes”
“Generate report every day at 02:00 UTC”
“Expire this trial in 14 days”

Unlike a simple queue of immediate work, schedulers must efficiently find **what becomes due now** among millions of future jobs, execute with workers, retry failures, and avoid double runs where possible.

What you will learn

Model one-off vs recurring jobs.
Store jobs with run_at indexes.
Dispatch due jobs to workers safely.
Use leases/visibility timeouts.
Handle cron calendars and catch-up policies.
Discuss sharding by time and tenant.

Words you need before we begin

Term	Plain English
Due job	Job whose `run_at` ≤ now.
Worker	Process that executes job payload.
Lease / visibility timeout	Temporary ownership so others do not run the same job.
Idempotent job	Safe if executed more than once.
Cron expression	Schedule description for recurring jobs.
Missed run policy	Skip vs catch up after downtime.
Dead-letter	Quarantine for permanently failing jobs.

Requirements

Functional

Schedule one-off job at timestamp
Schedule recurring job
Cancel / pause
At-least-once execution with retries

Non-functional

Low schedule/dispatch latency
Horizontal scale
Durable across restarts

High-level design

flowchart LR
  API[Scheduler API] --> DB[(Jobs store)]
  Disp[Dispatcher] --> DB
  Disp --> Q[Ready queue]
  Q --> W[Workers]
  W --> DB

Step-by-step design

Step 1 — Job record

job_id, tenant, payload, run_at, status, attempts, recurrence, idempotency_key.

Step 2 — Index for due scans

Index (status, run_at) or partition by time buckets. Avoid full table scans.

Step 3 — Dispatcher

Periodically (or via listen/notify) select due jobs, mark LEASED with owner + expiry, push to ready queue. Use UPDATE … WHERE status=READY AND run_at<=now LIMIT n RETURNING patterns or equivalent.

Step 4 — Workers

Execute; on success mark DONE; on failure retry with backoff or DLQ. Heartbeat lease for long jobs.

Step 5 — Recurring

On success, compute next run_at from cron; insert/update next occurrence. Do not enqueue infinite future rows—materialize next only.

Step 6 — Shard

Shard by tenant_id or hash(job_id) so dispatchers do not fight one hot table. Per-shard dispatcher with leader election optional.

Step 7 — Exactly-once talk

Be honest: at-least-once + idempotent handlers. Leases reduce doubles but clocks and crashes still need app idempotency.

Failure modes

Mode	Impact	Mitigation
Dispatcher lag	Late jobs	Scale dispatch; partition
Worker death mid-job	Re-run after lease	Idempotent effects
Thundering herd at :00	Spike	Jitter schedules
Poison payload	Infinite fail	Max attempts + DLQ
Clock skew	Early/late runs	NTP; trust DB time

Trade-offs

Choice	Benefit	Cost
DB as scheduler	Strong durability	Scan/contention care
Specialized systems (e.g. queues with delay)	Features	Ops/learn
Separate queue after due	Worker scale	Extra move

Common mistakes

SELECT * FROM jobs WHERE run_at < now every second unindexed.
Claiming exactly-once without idempotency.
Materializing 10 years of cron rows.
Single global lock for all jobs.
No tenant isolation.

Check your understanding

What makes scheduling different from a plain queue?
Why lease jobs?
How to store cron efficiently?
What is a missed-run policy?
Why jitter :00 crons?

Practice

Schema + indexes for 100M future jobs.
Sequence for worker crash mid-execution.
Design API for create/cancel.
Compare Quartz-like vs Kafka-delay approaches conceptually.
Mock interview.

Deeper production notes

Calendar edge cases

Time zones, DST, and “every 30th” calendars need explicit rules—mention complexity.

Observability

Track schedule lag (now - run_at), in-flight leases, DLQ depth.

Additional teaching scenarios

Scenario A — 10× peak

Which component saturates first? First mitigation?

Scenario B — dependency down 30 minutes

What still works? What degrades?

Scenario C — five-sentence wrap

Requirements, core mechanism, scale lever, failure mode, trade-off.

Revision summary

Scheduler = due index + lease + workers.
Recurring jobs materialize next run.
Prefer at-least-once + idempotency.
Shard and jitter to avoid hotspots.

Glossary

Term	Definition
Dispatcher	Component that moves due jobs to execution.
Lease	Temporary exclusive right to run a job.
Recurrence	Rule generating future run times.

Abbreviations and terminology

DLQ — Dead-letter queue
TTL — Time to live for leases
UTC — Coordinated Universal Time

What to learn next

Extra teaching notes

When you apply this lesson, write the user-visible success metric first, then the failure mode you fear most. Design the smallest mechanism that protects that metric under partial failure. Prefer explicit timeouts, idempotency, and ownership over adding more infrastructure boxes.

In interviews or design reviews, narrate assumptions, request paths, and trade-offs out loud. A correct-enough design with clear failure handling beats a buzzword diagram without numbers. Revisit the worked example and restate it for a different domain to prove you own the ideas, not the template wording.

Extra teaching notes

Interview and production field guide for this topic

Use this section as deliberate practice, not filler. Rewrite the worked example for a second domain you know well—fintech, education, logistics, or media. Keep the same skeleton: requirements, estimates, high-level diagram, request path, data model, scale lever, failure modes, and trade-offs. If you cannot fill every section without copying buzzwords, you do not yet own the design.

Numbers to force yourself to state

Always speak order-of-magnitude figures: peak QPS, storage growth per day, fan-out factor, connection counts, or queue depth. Wrong numbers that are explicit beat silent hand-waving. Correct the numbers when the interviewer or teammate challenges them; that is collaboration, not failure.

Failure minute

Set a timer for sixty seconds and list only failures: timeouts, duplicates, hot keys, dependency outages, bad deploys, and data corruption paths. For each, name detection and first mitigation. Designs that only describe the happy path are incomplete for production and weak in interviews.

Ownership and operability

Name the dashboard, the alert, the runbook section, and the team that pages. If any are blank, the system will train you during an incident. Prefer progressive delivery: canaries, flags, and rollback notes written before the change lands.

Consistency and retries

State whether the design assumes at-least-once delivery, whether handlers are idempotent, and where unique constraints live. Retries without idempotency are how double charges, double messages, and duplicate fan-out jobs appear. Timeouts without bounds are how thread pools die.

What good looks like in a review

A strong design review or interview answer clarifies scope, makes assumptions audible, draws a minimal path, deepens one or two bottlenecks, and closes with trade-offs and evolution. Use that bar on design-job-scheduler round 0 every time you revisit it.

Interview and production field guide for this topic

Numbers to force yourself to state

Failure minute

Ownership and operability

Consistency and retries

What good looks like in a review

FAQ from first-time learners

Q: Is cron on one server enough?
A: For tiny systems yes; it is a SPOF and does not scale—say when to evolve.

Q: Database or Redis ZSET by score=run_at?
A: Both are interview-valid; discuss durability and blocking pops.

Track: Distributed Systems

Next: Design a Notification Service — Push, Email, SMS

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab