Our backups stopped on a Sunday. We found out 11 weeks later — from a restore test, not an alert.

Description and use

Silent cron job failures were detected using heartbeats, output alerts, and a schedule inventory, preventing data loss.

Problem

Critical cron jobs silently stopped running, causing undetected backup failures for weeks.

Outcome

Heartbeat and output monitoring uncovered silent backup failures, preventing data loss and improving reliability.

Workflow

• Identify critical cron jobs (backups, syncs, digests).
• Add a heartbeat line (e.g., curl a monitor URL) to each job on success.
• Create a separate cron that checks each heartbeat timestamp against a threshold and alerts if stale.
• Log job output counts (rows, files, bytes) and alert when zero.
• Maintain an inventory of all scheduled tasks with owners and purpose.
• Test by disabling a job and verifying the silence alert fires.

Evidence

The backup just... wasn't there.

Source

The fix that catches silence: heartbeats.

Source

The fix that catches uselessness: alert on output.

Source

The fix that catches drift: own the schedule.

Source