If the Job Cannot Show What It Did, Somebody Will Do It Again

Description and use

Automated provisioning jobs reported success without verifying actual changes, causing hidden failures and manual reviews. Adding state verification and logging eliminated manual checks and exposed drift.

Problem

Automated provisioning jobs reported success without confirming actual changes, leading to undetected failures and manual rework.

Outcome

Implemented state verification and logging reduced manual review time, surfaced hidden failures, and enabled automated drift detection.

Workflow

• Run provisioning job
• Script performs API calls
• Script checks API acceptance, writes green tick
• Engineer manually reviews console to confirm settings

Evidence

Our automation told us that it had run. It did not tell us what it had changed, and an exit code of zero is a statement about a script, not about a system.

Source

A daily reconciliation compares intent against reality for the whole estate, so drift is reported rather than discovered.

Source