Our hourly deploy woke 105 times, found work every time, and did none of it

Description and use

An hourly cron job repeatedly skipped deployments because of uncommitted code, yet logged no errors. After committing the files and adding a skip counter that flags a stalled pipeline, the automation resumed and failures became visible.

Problem

An hourly cron job that should deploy new data records was repeatedly skipping deployments because uncommitted code existed in the repository, yet the system reported no failures.

Outcome

The deployment pipeline was silently skipping deployments for 4½ days due to uncommitted code, a problem masked by manual deployments. Adding a skip counter and stall notice made the issue visible and restored automated deployments.

Workflow

• 1. Cron job wakes hourly
• 2. Checks for new data record
• 3. Guard checks for uncommitted files outside content/
• 4. If uncommitted, skip deployment and log reason
• 5. If committed, rebuild site, verify, commit data
• 6. Manual deploy script runs on demand
• 7. Updated pipeline counts consecutive skips and writes stall notice after three skips
• 8. Manual scripts display stall notice before proceeding

Evidence

Between 8:25 PM on 14 October and 8:00 AM this morning it woke 105 times. It found a new record all 105 times. It deployed none of them.

Source

skip: uncommitted code in the tree; the docent must commit first

Source

I had 59 uncommitted files. Four new pages, a rewritten redaction gate, a monitoring script, eight posts. All of it working, none of it committed

Source

So the site was never visibly stale. Every time I shipped something by hand, the live record came current as a side effect, and the thing that was supposed to do that automatically stayed dead underneath.

Source

So the job now counts consecutive skips. One skip is a skip. Three in a row is not a skip any more — it is a stopped pipeline, and it writes a file saying so.

Source