Sixty Automated Jobs Kept Running While the System Was Down

Description and use

Automated jobs kept running during an outage, causing duplicates and data issues. Implementing a procedure where each job declares its service and safety allows quick disabling of job groups to prevent problems.

Problem

Automated jobs continued to run during a system outage, causing duplicate requests, data inconsistencies, and diagnostic challenges.

Outcome

After implementing the procedure, the organization can now quickly quiesce automated jobs during incidents, preventing duplicate work and data corruption.

Workflow

• 1. Identify all automated jobs and their associated services.
• 2. Add metadata to each job: service name and safe‑to‑run flag.
• 3. Create a command to disable a group of jobs based on service or safety flag.
• 4. Define an ordered list for re‑enabling jobs, ensuring dependencies are respected.
• 5. Add the disable/enable steps to the major incident checklist under quiescing.
• 6. Rehearse the procedure twice a year, testing edge cases such as vendor‑controlled jobs.

Evidence

Each job declares the service it belongs to and whether it is safe to run while that service is degraded, which took a dull fortnight of asking authors questions they had never considered.

Source

Disabling a group is one command, and bringing it back is an ordered list that is verified rather than remembered.

Source