The 2AM Page Is a Design Choice: 7 Safeguards That Silenced Our On-Call Phone

说明与用途

实施七项安全措施将夜间警报减少了40–60%，实现了常见故障的自动化，并提升了值班效率。

问题

On-call teams receive frequent 2AM pages for failures that could be automatically handled, causing unnecessary human intervention.

结果

The seven safeguards collectively cut 2AM page volume by 40–60%, shifted decision‑making from humans to automated systems, and ensured timely, accountable incident resolution.

工作流

• 1. Pull last 90 days of alerts and identify top 3 recurring pages (disk full, memory leak, dead worker).
• 2. Write 5‑line auto‑remediation scripts for each and schedule them via cron, logging each action.
• 3. Add a direct runbook link to every alert payload to eliminate search time.
• 4. Define three severity tiers: page now, business hours, log only, and audit monthly.
• 5. Configure sleep‑based suppression windows for deploys and batch jobs on tagged services.
• 6. Schedule a 20‑minute restore drill monthly to verify backup integrity.
• 7. Implement a three‑tier escalation policy with defined time thresholds and incident commander assignment.
• 8. Require each incident postmortem to have one owner and one due date for the fix.

证据

This alone kills 40–60% of page volume.

来源

If a responder has to find the runbook at 2AM, you don't have a runbook, you have a scavenger hunt.

来源