Runbook Rot: Three Fingerprints of Stale CI Docs and the Check That Catches Each

Description and use

Automated CI checks for runbook freshness detect timestamp gaps, orphaned identifiers, and procedural verbs, preventing costly night‑time errors.

Stale CI runbooks cause costly errors when the documented pipeline no longer matches the live environment.

Runbooks are now automatically validated for freshness, reducing 3 a.m. manual interventions and preventing costly errors.

Extract last content edit of the doc and compare it to the last edit of its subject to detect timestamp gaps.
Extract backticked identifiers from the runbook and verify each against the repository to find orphaned referents.
Search for procedural verbs (e.g., "ask", "ping", "check") and flag them as work tickets.
Add a front‑matter header with as_of, scope, cadence, and owner to each runbook.
Run a CI job that calculates lag = now() - as_of and fails the build if lag exceeds cadence.
When a failure occurs, bump as_of only after a human verifies the steps against the live system.
Repeat the checks on a scheduled cadence to maintain freshness.

The document's last meaningful edit predates the last change to the thing it describes.

The check: extract identifiers from the doc and verify each against ground truth

The check is a grep, and every hit is a work ticket

Then a one-page CI job computes lag = now() - as_of against cadence and fails the build at 2× expected interval