My Pipeline Went Silent for Five Hours — Every Job Failed the Same Quiet Way

Description and use

A missing network share caused all scheduled jobs to silently fail. The fix added timeouts to network calls, ran jobs locally, used local-first logging with delta shipping, and introduced a watchdog that alerts on share absence.

Problem

A network share mounted at /Volumes/share went missing, causing all scheduled jobs that depended on it to silently fail for five hours without generating alerts.

Outcome

Implemented timeouts on network calls, removed single point of failure by running jobs locally, added local-first logging with delta shipping, and introduced a watchdog that alerts on share absence, preventing silent outages.

Workflow

• 1. Detect missing mount via timestamp 02:35:43 [TRANSITION] OK -> UNMOUNTED
• 2. Identify that all jobs’ working directories were on /Volumes/share
• 3. Realize cron jobs fail silently because MAILTO="" and output is lost
• 4. Find auto‑mount helper hung without timeout, blocking recovery
• 5. Implement run_with_timeout wrapper for network calls
• 6. Move job working directories to local disk
• 7. Implement local-first logging with a delta shipper to NAS
• 8. Add watchdog that alerts when share is absent for >15 minutes and on recovery

Evidence

Every scheduled job had one thing in common: its working directory lived on a network share mounted at /Volumes/share.

Source

The classic line MAILTO="" in the crontab — I set it years ago to stop noise — means a job that dies produces no message at all.

Source

That osascript call has no timeout. Last night it hung — and stayed hung for five hours.

Source

The remount logic now wraps the blocking call in a hard timeout…

Source

Jobs run from local disk. The working directory moved off the network share entirely.

Source

A tiny watcher runs every five minutes and only speaks on a state change…

Source