Two schedulers, one question: how we keep a 150-job small-business ops platform reliable

说明与用途

两套调度器通过区分基础设施任务和业务任务，并使用统一仪表盘、日志、重试和陈旧性警报，确保150+任务的可靠运行。

问题

Small‑business operations involve many repetitive tasks with hard deadlines (SMS confirmations, reminders, invoices, etc.). Missing any task can lead to client dissatisfaction.

结果

The platform now reliably executes 150+ scheduled jobs, with infrastructure tasks surviving app crashes and business jobs avoiding duplicate execution. Monitoring detects failures before clients notice.

工作流

• 1. Classify each job as "needs to run when app is down" or "does not need to run when app is down".
• 2. Schedule infrastructure jobs (e.g., backups, watchdog) in the OS task scheduler (Windows Task Scheduler).
• 3. Schedule business jobs (e.g., reminders, reports) in the in‑app scheduler (Python schedule library).
• 4. Deploy a read‑only dashboard that lists both scheduler rings with last/next run times.
• 5. Ensure every job logs, retries, and reports failures.
• 6. Monitor staleness by checking if a job performed real work, not just exited successfully.
• 7. Use a watchdog to restart the app automatically, preventing duplicate job runs.
• 8. Send plain‑English alerts for routine successes or failures.

证据

Every recurring job lives in one of two places, and one question decides which: Does this job need to run when the app itself is not running?

来源

A backup job living inside the app does nothing on exactly the night the app has crashed, which is the night you needed it.

来源

A reminder job moved out of the app to "make it independent" loses the shared state and locks it was quietly relying on, and starts double‑sending.

来源