Sixty Automated Jobs Kept Running While the System Was Down

说明与用途

自动化作业在系统故障期间继续运行，导致重复和数据问题。实施每个作业声明其服务和安全性的程序，可快速禁用作业组以防止问题。

问题

Automated jobs continued to run during a system outage, causing duplicate requests, data inconsistencies, and diagnostic challenges.

结果

After implementing the procedure, the organization can now quickly quiesce automated jobs during incidents, preventing duplicate work and data corruption.

工作流

• 1. Identify all automated jobs and their associated services.
• 2. Add metadata to each job: service name and safe‑to‑run flag.
• 3. Create a command to disable a group of jobs based on service or safety flag.
• 4. Define an ordered list for re‑enabling jobs, ensuring dependencies are respected.
• 5. Add the disable/enable steps to the major incident checklist under quiescing.
• 6. Rehearse the procedure twice a year, testing edge cases such as vendor‑controlled jobs.

证据

Each job declares the service it belongs to and whether it is safe to run while that service is degraded, which took a dull fortnight of asking authors questions they had never considered.

来源

Disabling a group is one command, and bringing it back is an ordered list that is verified rather than remembered.

来源