Two schedulers, one question: how we keep a 150-job small-business ops platform reliable

Описание и применение

Две системы планировщиков обеспечивают надёжную работу 150+ задач в платформе малого бизнеса, разделяя инфраструктурные и бизнес‑задачи и применяя мониторинг и watchdog.

Задача

Small‑business operations involve many repetitive tasks with hard deadlines (SMS confirmations, reminders, invoices, etc.). Missing any task can lead to client dissatisfaction.

Результат

The platform now reliably executes 150+ scheduled jobs, with infrastructure tasks surviving app crashes and business jobs avoiding duplicate execution. Monitoring detects failures before clients notice.

Рабочий процесс

• 1. Classify each job as "needs to run when app is down" or "does not need to run when app is down".
• 2. Schedule infrastructure jobs (e.g., backups, watchdog) in the OS task scheduler (Windows Task Scheduler).
• 3. Schedule business jobs (e.g., reminders, reports) in the in‑app scheduler (Python schedule library).
• 4. Deploy a read‑only dashboard that lists both scheduler rings with last/next run times.
• 5. Ensure every job logs, retries, and reports failures.
• 6. Monitor staleness by checking if a job performed real work, not just exited successfully.
• 7. Use a watchdog to restart the app automatically, preventing duplicate job runs.
• 8. Send plain‑English alerts for routine successes or failures.

Доказательства

Every recurring job lives in one of two places, and one question decides which: Does this job need to run when the app itself is not running?

Источник

A backup job living inside the app does nothing on exactly the night the app has crashed, which is the night you needed it.

Источник

A reminder job moved out of the app to "make it independent" loses the shared state and locks it was quietly relying on, and starts double‑sending.

Источник