My Pipeline Went Silent for Five Hours — Every Job Failed the Same Quiet Way

说明与用途

网络共享挂载失效导致所有计划任务静默失败。解决方案包括为网络调用添加超时、将作业迁移到本地磁盘、使用本地优先日志并增量同步，以及引入监控缺失共享的 watchdog。

问题

A network share mounted at /Volumes/share went missing, causing all scheduled jobs that depended on it to silently fail for five hours without generating alerts.

结果

Implemented timeouts on network calls, removed single point of failure by running jobs locally, added local-first logging with delta shipping, and introduced a watchdog that alerts on share absence, preventing silent outages.

工作流

• 1. Detect missing mount via timestamp 02:35:43 [TRANSITION] OK -> UNMOUNTED
• 2. Identify that all jobs’ working directories were on /Volumes/share
• 3. Realize cron jobs fail silently because MAILTO="" and output is lost
• 4. Find auto‑mount helper hung without timeout, blocking recovery
• 5. Implement run_with_timeout wrapper for network calls
• 6. Move job working directories to local disk
• 7. Implement local-first logging with a delta shipper to NAS
• 8. Add watchdog that alerts when share is absent for >15 minutes and on recovery

证据

Every scheduled job had one thing in common: its working directory lived on a network share mounted at /Volumes/share.

来源

The classic line MAILTO="" in the crontab — I set it years ago to stop noise — means a job that dies produces no message at all.

来源

That osascript call has no timeout. Last night it hung — and stayed hung for five hours.

来源

The remount logic now wraps the blocking call in a hard timeout…

来源

Jobs run from local disk. The working directory moved off the network share entirely.

来源

A tiny watcher runs every five minutes and only speaks on a state change…

来源