Runbook Rot: Three Fingerprints of Stale CI Docs and the Check That Catches Each

Описание и применение

Автоматический контроль актуальности runbook‑ов в CI позволяет выявлять устаревшие описания, удалённые идентификаторы и устаревшие команды, предотвращая ошибки в ночных сменах.

Задача

Stale CI runbooks cause costly errors when the documented pipeline no longer matches the live environment.

Результат

Runbooks are now automatically validated for freshness, reducing 3 a.m. manual interventions and preventing costly errors.

Рабочий процесс

• Extract last content edit of the doc and compare it to the last edit of its subject to detect timestamp gaps.
• Extract backticked identifiers from the runbook and verify each against the repository to find orphaned referents.
• Search for procedural verbs (e.g., "ask", "ping", "check") and flag them as work tickets.
• Add a front‑matter header with as_of, scope, cadence, and owner to each runbook.
• Run a CI job that calculates lag = now() - as_of and fails the build if lag exceeds cadence.
• When a failure occurs, bump as_of only after a human verifies the steps against the live system.
• Repeat the checks on a scheduled cadence to maintain freshness.

Доказательства

The document's last meaningful edit predates the last change to the thing it describes.

Источник

The check: extract identifiers from the doc and verify each against ground truth

Источник

The check is a grep, and every hit is a work ticket

Источник

Then a one-page CI job computes lag = now() - as_of against cadence and fails the build at 2× expected interval

Источник