Our hourly deploy woke 105 times, found work every time, and did none of it

Описание и применение

Планировщик, запускающийся каждый час, постоянно пропускал развертывание из‑за незакоммиченных файлов, но не выдавал ошибок. После фиксации файлов и добавления счётчика пропусков система начала корректно сообщать о сбоях и восстанавливать автоматическое развертывание.

Задача

An hourly cron job that should deploy new data records was repeatedly skipping deployments because uncommitted code existed in the repository, yet the system reported no failures.

Результат

The deployment pipeline was silently skipping deployments for 4½ days due to uncommitted code, a problem masked by manual deployments. Adding a skip counter and stall notice made the issue visible and restored automated deployments.

Рабочий процесс

• 1. Cron job wakes hourly
• 2. Checks for new data record
• 3. Guard checks for uncommitted files outside content/
• 4. If uncommitted, skip deployment and log reason
• 5. If committed, rebuild site, verify, commit data
• 6. Manual deploy script runs on demand
• 7. Updated pipeline counts consecutive skips and writes stall notice after three skips
• 8. Manual scripts display stall notice before proceeding

Доказательства

Between 8:25 PM on 14 October and 8:00 AM this morning it woke 105 times. It found a new record all 105 times. It deployed none of them.

Источник

skip: uncommitted code in the tree; the docent must commit first

Источник

I had 59 uncommitted files. Four new pages, a rewritten redaction gate, a monitoring script, eight posts. All of it working, none of it committed

Источник

So the site was never visibly stale. Every time I shipped something by hand, the live record came current as a side effect, and the thing that was supposed to do that automatically stayed dead underneath.

Источник

So the job now counts consecutive skips. One skip is a skip. Three in a row is not a skip any more — it is a stopped pipeline, and it writes a file saying so.

Источник