I stopped losing background jobs to sleep mode: a patient queue with leases

Описание и применение

Система slothqueue использует аренду и отложенные повторные попытки, чтобы гарантировать, что фоновые задачи на ноутбуке не теряются и не дублируются при переходе в спящий режим или падении процесса.

Задача

Background jobs on a laptop are lost or duplicated when the machine sleeps or the process crashes, because the job state does not distinguish between a slow job and a dead one.

Результат

The slothqueue implementation prevents job loss or duplication on laptops by using leases and scheduled retries, enabling background jobs to survive sleep mode and crashes without extra infrastructure.

Рабочий процесс

• 1. Worker calls claim() to reserve a job; the job’s lease_until is set to now + lease_duration and attempts is incremented.
• 2. Worker performs the job, periodically calling heartbeat() to extend the lease while the work is genuinely in progress.
• 3. Upon successful completion, worker calls complete() to mark the job as done and record a message.
• 4. If the worker crashes or the machine sleeps before completion, the lease expires; the next claim() will re‑issue the job with attempts == 2 and a reclaimed event.
• 5. If a job fails, the worker schedules the next attempt by setting not_before = now + 2**attempts (capped), so claim() only sees jobs whose not_before has passed.
• 6. When attempts reach max_attempts, the job is marked failed and no longer retried.
• 7. The system runs with a single SQLite database file, no network or broker, and no distributed workers.
• 8. Behaviour is verified by nine unit tests that inject a clock to test lease expiry, backoff, and failure handling without real sleeping.

Доказательства

If the process dies between claim and complete, nothing needs to notice. The lease simply expires, and the next claim() re-issues the job with attempts == 2 and a reclaimed event in its history.

Источник

failure schedules 2, 4, 8 seconds, capped, then lands in failed;

Источник

python -m unittest discover -s tests -v # 9 tests, ~0.3s

Источник

no distributed workers (a laptop doesn't need them — run two queues in two folders if you want two lanes); no parallelism; no dependencies at all — sqlite3 and subprocess from the standard library.

Источник