Our backups stopped on a Sunday. We found out 11 weeks later — from a restore test, not an alert.

Описание и применение

Скрытые сбои cron‑задач выявлены с помощью сигналов сердцебиения, логов и инвентаризации расписаний, что предотвратило потерю данных.

Задача

Critical cron jobs silently stopped running, causing undetected backup failures for weeks.

Результат

Heartbeat and output monitoring uncovered silent backup failures, preventing data loss and improving reliability.

Рабочий процесс

• Identify critical cron jobs (backups, syncs, digests).
• Add a heartbeat line (e.g., curl a monitor URL) to each job on success.
• Create a separate cron that checks each heartbeat timestamp against a threshold and alerts if stale.
• Log job output counts (rows, files, bytes) and alert when zero.
• Maintain an inventory of all scheduled tasks with owners and purpose.
• Test by disabling a job and verifying the silence alert fires.

Доказательства

The backup just... wasn't there.

Источник

The fix that catches silence: heartbeats.

Источник

The fix that catches uselessness: alert on output.

Источник

The fix that catches drift: own the schedule.

Источник