Sixty Automated Jobs Kept Running While the System Was Down

Описание и применение

Автоматизированные задания продолжали работать во время сбоя, создавая дубли и ошибки. Внедрение процедуры, где каждое задание объявляет свою службу и безопасность, позволяет быстро отключать группы заданий и предотвращать проблемы.

Задача

Automated jobs continued to run during a system outage, causing duplicate requests, data inconsistencies, and diagnostic challenges.

Результат

After implementing the procedure, the organization can now quickly quiesce automated jobs during incidents, preventing duplicate work and data corruption.

Рабочий процесс

• 1. Identify all automated jobs and their associated services.
• 2. Add metadata to each job: service name and safe‑to‑run flag.
• 3. Create a command to disable a group of jobs based on service or safety flag.
• 4. Define an ordered list for re‑enabling jobs, ensuring dependencies are respected.
• 5. Add the disable/enable steps to the major incident checklist under quiescing.
• 6. Rehearse the procedure twice a year, testing edge cases such as vendor‑controlled jobs.

Доказательства

Each job declares the service it belongs to and whether it is safe to run while that service is degraded, which took a dull fortnight of asking authors questions they had never considered.

Источник

Disabling a group is one command, and bringing it back is an ordered list that is verified rather than remembered.

Источник