My Auto-Remediation Bot Fixed the Wrong Service. Here’s What That Taught Me About Real SRE

Description and use

The auto‑remediation bot initially misidentified the root cause, acted on the wrong service, and falsely reported recovery. After fixing alert routing, adding a static dependency map, and enforcing real Prometheus data for success, the system now correctly diagnoses failures and produces accurate post‑mortems.

Problem

An auto‑remediation controller mis‑identified the root cause, acted on the wrong service, and falsely reported recovery, exposing gaps in alert routing, dependency awareness, and success criteria.

Outcome

The auto‑remediation system was hardened to correctly identify root causes across service dependencies, avoid false recoveries, and produce reliable post‑mortems, demonstrating the necessity of real‑failure testing and transparent documentation.

Workflow

• 1. Inject chaos to kill inventory pods via Chaos Mesh
• 2. Verify Alertmanager fires alerts with correct labels
• 3. Ensure Prometheus routes alerts to controller by adding missing namespace label
• 4. Update controller to use static dependency map to trace root cause
• 5. Remove blind restart fallback and enforce Prometheus data for recovery
• 6. Validate logic with synthetic alerts before real chaos
• 7. Run real chaos test and confirm correct remediation
• 8. Generate automated post‑mortem with Prometheus data
• 9. Review metrics for confirmed/unconfirmed recoveries and fix bucket logic

Evidence

Problem 1 — the alert never reached the controller.

Source

Problem 2 — the controller acted on the wrong service.

Source

Problem 3 — it lied about recovering.

Source