I stopped losing background jobs to sleep mode: a patient queue with leases

Description and use

The slothqueue system uses leases and delayed retries to ensure that background jobs on a laptop are not lost or duplicated when the machine sleeps or the process crashes.

Problem

Background jobs on a laptop are lost or duplicated when the machine sleeps or the process crashes, because the job state does not distinguish between a slow job and a dead one.

Outcome

The slothqueue implementation prevents job loss or duplication on laptops by using leases and scheduled retries, enabling background jobs to survive sleep mode and crashes without extra infrastructure.

Workflow

• 1. Worker calls claim() to reserve a job; the job’s lease_until is set to now + lease_duration and attempts is incremented.
• 2. Worker performs the job, periodically calling heartbeat() to extend the lease while the work is genuinely in progress.
• 3. Upon successful completion, worker calls complete() to mark the job as done and record a message.
• 4. If the worker crashes or the machine sleeps before completion, the lease expires; the next claim() will re‑issue the job with attempts == 2 and a reclaimed event.
• 5. If a job fails, the worker schedules the next attempt by setting not_before = now + 2**attempts (capped), so claim() only sees jobs whose not_before has passed.
• 6. When attempts reach max_attempts, the job is marked failed and no longer retried.
• 7. The system runs with a single SQLite database file, no network or broker, and no distributed workers.
• 8. Behaviour is verified by nine unit tests that inject a clock to test lease expiry, backoff, and failure handling without real sleeping.

Evidence

If the process dies between claim and complete, nothing needs to notice. The lease simply expires, and the next claim() re-issues the job with attempts == 2 and a reclaimed event in its history.

Source

failure schedules 2, 4, 8 seconds, capped, then lands in failed;

Source

python -m unittest discover -s tests -v # 9 tests, ~0.3s

Source

no distributed workers (a laptop doesn't need them — run two queues in two folders if you want two lanes); no parallelism; no dependencies at all — sqlite3 and subprocess from the standard library.

Source