Our backups stopped on a Sunday. We found out 11 weeks later — from a restore test, not an alert.

说明与用途

通过心跳、输出警报和调度清单检测到隐藏的 cron 任务失败，防止了数据丢失。

问题

Critical cron jobs silently stopped running, causing undetected backup failures for weeks.

结果

Heartbeat and output monitoring uncovered silent backup failures, preventing data loss and improving reliability.

工作流

• Identify critical cron jobs (backups, syncs, digests).
• Add a heartbeat line (e.g., curl a monitor URL) to each job on success.
• Create a separate cron that checks each heartbeat timestamp against a threshold and alerts if stale.
• Log job output counts (rows, files, bytes) and alert when zero.
• Maintain an inventory of all scheduled tasks with owners and purpose.
• Test by disabling a job and verifying the silence alert fires.

证据

The backup just... wasn't there.

来源

The fix that catches silence: heartbeats.

来源

The fix that catches uselessness: alert on output.

来源

The fix that catches drift: own the schedule.

来源