· 2 min read
Lease and fencing: how a queue neither loses a job nor lets a zombie finish it twice
A worker can crash, stall, or wake up after its job was taken by someone else. How the lease and the lease token in job-queue deal with that on SQLite.
A background queue has two nasty situations. A worker dies in the middle of a job and the job stays "running" forever. Or a worker simply freezes for a while (a GC pause, the network), the job is handed to another worker, and then the first one wakes up and writes its result over someone else's.
Lease
Claiming a job grants a lease: a lease_until deadline (30 seconds by default). While the handler runs, the worker extends the lease with a heartbeat every third of the term. If the process dies, the heartbeat stops, the lease expires, and the "reaper" returns the job to the queue with backoff or, if attempts are exhausted, to dead.
Claiming with one statement
Picking the job and granting the lease is a single SQL statement:
UPDATE jobs SET status='running', attempts=attempts+1,
lease_until=?, worker=?, token=?, started_at=?
WHERE id=(SELECT id FROM jobs WHERE status='pending' AND run_at<=?
ORDER BY priority DESC, run_at ASC, id ASC LIMIT 1)
RETURNING ...So two workers, even from different processes on one database file, cannot get the same job.
Fencing
Every claim stores a unique token. Extending, completing and failing run with WHERE status='running' AND token=?. If the job was already taken over, the condition does not match, and the zombie worker gets ErrLeaseLost instead of corrupting the result.
What it does not give you
The guarantee stays at-least-once: after a lease expires, a handler may be called again, so it must be idempotent. The queue does not promise exactly-once. Details and limitations are in the project README.