The 2AM Page Is a Design Choice: 7 Safeguards That Silenced Our On-Call Phone

Описание и применение

Внедрение семи защитных мер сократило ночные оповещения на 40–60 %, автоматизировало типичные сбои и повысило эффективность реагирования.

Задача

On-call teams receive frequent 2AM pages for failures that could be automatically handled, causing unnecessary human intervention.

Результат

The seven safeguards collectively cut 2AM page volume by 40–60%, shifted decision‑making from humans to automated systems, and ensured timely, accountable incident resolution.

Рабочий процесс

• 1. Pull last 90 days of alerts and identify top 3 recurring pages (disk full, memory leak, dead worker).
• 2. Write 5‑line auto‑remediation scripts for each and schedule them via cron, logging each action.
• 3. Add a direct runbook link to every alert payload to eliminate search time.
• 4. Define three severity tiers: page now, business hours, log only, and audit monthly.
• 5. Configure sleep‑based suppression windows for deploys and batch jobs on tagged services.
• 6. Schedule a 20‑minute restore drill monthly to verify backup integrity.
• 7. Implement a three‑tier escalation policy with defined time thresholds and incident commander assignment.
• 8. Require each incident postmortem to have one owner and one due date for the fix.

Доказательства

This alone kills 40–60% of page volume.

Источник

If a responder has to find the runbook at 2AM, you don't have a runbook, you have a scavenger hunt.

Источник