On-call rotations: stop using a spreadsheet someone forgot to update

说明与用途

从手动电子表格排班转向自动化的 on‑call 系统，制定清晰的轮换规则、升级路径和补偿措施，使用 PagerDuty、Opsgenie 或简单的 cron+Slack。

问题

Manual on‑call schedules in spreadsheets cause coverage gaps, burnout, lack of escalation paths, and holiday blindness.

结果

Teams transition from unreliable spreadsheet schedules to automated, transparent on‑call systems, reducing incidents, preventing burnout, and ensuring clear escalation paths.

工作流

• Define rotation rules: primary/secondary, weekly rotation, no more than 2 consecutive weeks per engineer, handoff at Monday 10am local.
• Set escalation policy: primary 5 min, secondary 10 min, manager 15 min, CTO 30 min, all tracked automatically.
• Implement follow‑the‑sun for global teams: APAC 00:00‑08:00 UTC, EMEA 08:00‑16:00 UTC, Americas 16:00‑00:00 UTC, each with primary and secondary.
• Establish compensation and recovery: weekly on‑call bonus, day off after night‑time incident, no meetings after major incidents.
• Choose a tool: PagerDuty (best overall), Opsgenie (cheaper), cron+Slack (small teams), or custom Python script for full control.
• Create a template: include primary/secondary contacts, escalation steps, handoff time, and runbook updates.
• Schedule handoffs: review pending incidents, update runbook with learnings.
• Avoid common mistakes: don’t assign newest engineer as primary, don’t skip handoff, plan holiday coverage 2 weeks ahead, make on‑call mandatory.
• Provide ready‑to‑use templates and scripts via Ops Starter Kit.

证据

Coverage gaps: Nobody realizes the schedule is wrong until an incident hits

来源

Primary has 5 minutes to acknowledge; Secondary gets paged at 10 minutes; Manager notified at 15 minutes; All tracked automatically

来源

PagerDuty: Best overall, but expensive for small teams; Opsgenie: Good alternative, cheaper; Simple cron + Slack: For teams under 5 people, this works fine; Custom Python script: If you want full control

来源