My Pipeline Went Silent for Five Hours — Every Job Failed the Same Quiet Way

Описание и применение

Сеть‑шар, используемый как рабочий каталог, исчез, и все запланированные задачи тихо упали без сигналов. Решение включало таймауты для сетевых вызовов, перемещение работы на локальный диск, локальное логирование с последующей синхронизацией и watchdog, оповещающий об отсутствии шар‑а.

Задача

A network share mounted at /Volumes/share went missing, causing all scheduled jobs that depended on it to silently fail for five hours without generating alerts.

Результат

Implemented timeouts on network calls, removed single point of failure by running jobs locally, added local-first logging with delta shipping, and introduced a watchdog that alerts on share absence, preventing silent outages.

Рабочий процесс

• 1. Detect missing mount via timestamp 02:35:43 [TRANSITION] OK -> UNMOUNTED
• 2. Identify that all jobs’ working directories were on /Volumes/share
• 3. Realize cron jobs fail silently because MAILTO="" and output is lost
• 4. Find auto‑mount helper hung without timeout, blocking recovery
• 5. Implement run_with_timeout wrapper for network calls
• 6. Move job working directories to local disk
• 7. Implement local-first logging with a delta shipper to NAS
• 8. Add watchdog that alerts when share is absent for >15 minutes and on recovery

Доказательства

Every scheduled job had one thing in common: its working directory lived on a network share mounted at /Volumes/share.

Источник

The classic line MAILTO="" in the crontab — I set it years ago to stop noise — means a job that dies produces no message at all.

Источник

That osascript call has no timeout. Last night it hung — and stayed hung for five hours.

Источник

The remount logic now wraps the blocking call in a hard timeout…

Источник

Jobs run from local disk. The working directory moved off the network share entirely.

Источник

A tiny watcher runs every five minutes and only speaks on a state change…

Источник