Design a job scheduler with DAGs
A million five-minute jobs a day across a hundred thousand DAGs, on ten thousand workers. The round turns on the word once: a scheduler that owns each DAG's state leases every job to one worker, and a fencing token keeps a worker whose lease ran out from writing anything.
The round this design follows
How does a scheduler make sure a job is not run twice at the same time?
It leases the job: a conditional update in the store moves the job from ready to leased only if nobody holds a live lease, and hands the worker a fencing token one higher than the last. The worker renews the lease every thirty seconds. A lease alone is not enough, because a paused worker can wake after its lease ran out and keep going; the token fixes that, since the store and the result store refuse any write carrying a token lower than the one on record.
What is a fencing token?
A number that goes up by one every time a job is leased to a new worker. The worker sends it with every write, and whatever it writes to keeps the highest token it has seen and refuses anything lower. It turns "only one worker should be running" into "only one worker's writes are accepted", which is a promise the design can actually keep when clocks drift and processes pause.
What happens when the scheduler itself dies?
Each shard of DAGs has a leader and a standby, and leadership is itself a lease with an epoch number. When the leader stops renewing, the standby takes the lease within about a minute and runs the tick it missed, because the tick asks which DAGs have a next run time in the past, not which minute it is. Running jobs are not touched; only new starts wait. A unique index on the DAG and its scheduled time means a tick that runs twice creates one run.
How do you stop one huge DAG from taking every worker?
Cap how much of the pool one DAG may hold, say thirty percent. A fan-out of ten thousand five-minute jobs on three thousand workers takes about seventeen minutes instead of five, and every other DAG keeps starting its jobs within the minute. The cap does not add capacity; it decides which DAG waits, and the one that waits is the one that asked for the whole pool.
Why not have workers poll a jobs table with row locks?
It is the right design for a few thousand jobs a day and the wrong one here. Ten thousand idle workers asking every five seconds is two thousand locking queries a second on one table, most of them finding nothing. And a row lock only lasts the claiming transaction, so covering a five-minute job needs a lease column and a sweeper anyway, which is the coordinator design without an owner for the DAG state.