My Auto-Remediation Bot Fixed the Wrong Service. Here’s What That Taught Me About Real SRE

说明与用途

自动修复机器人最初错误识别根本原因，错误地操作了服务，并错误地报告恢复。通过修复警报路由、添加静态依赖映射并强制使用真实 Prometheus 数据进行成功判断，系统现在能够正确诊断故障并生成准确的事后报告。

问题

An auto‑remediation controller mis‑identified the root cause, acted on the wrong service, and falsely reported recovery, exposing gaps in alert routing, dependency awareness, and success criteria.

结果

The auto‑remediation system was hardened to correctly identify root causes across service dependencies, avoid false recoveries, and produce reliable post‑mortems, demonstrating the necessity of real‑failure testing and transparent documentation.

工作流

• 1. Inject chaos to kill inventory pods via Chaos Mesh
• 2. Verify Alertmanager fires alerts with correct labels
• 3. Ensure Prometheus routes alerts to controller by adding missing namespace label
• 4. Update controller to use static dependency map to trace root cause
• 5. Remove blind restart fallback and enforce Prometheus data for recovery
• 6. Validate logic with synthetic alerts before real chaos
• 7. Run real chaos test and confirm correct remediation
• 8. Generate automated post‑mortem with Prometheus data
• 9. Review metrics for confirmed/unconfirmed recoveries and fix bucket logic

证据

Problem 1 — the alert never reached the controller.

来源

Problem 2 — the controller acted on the wrong service.

来源

Problem 3 — it lied about recovering.

来源