My self-healing homelab saved 17 seconds. Here’s why I kept it

说明与用途

在家庭实验室中实现的自动化修复链将停机时间缩短了17秒，防止了级联重启，并记录了所有操作以供审计。

问题

A power cut caused the reverse proxy to fail, delaying alerts and leaving sites inaccessible until the issue was discovered hours later.

结果

The automated chain restored services 17 seconds faster than a human responder, prevented cascading restarts, and logged all actions for audit.

工作流

• 1. Prometheus detects service health via probes.
• 2. Alertmanager forwards alerts to n8n via webhook.
• 3. n8n drops stale alerts and forwards the current alert to the policy gate.
• 4. The policy gate evaluates the alert against a closed catalogue of actions and returns a verdict (act or refuse).
• 5. If act, the gate sends the action to the executor (rb-exec) which runs a single command (e.g., docker restart).
• 6. The gate re‑checks Prometheus for proof of recovery up to 120 s after the action.
• 7. If recovery fails, the action is logged as a failure and the user is paged.
• 8. Guardrails (circuit breaker, no‑op counting, failure thresholds) prevent runaway remediation.
• 9. A canary alert periodically verifies the chain’s health by running a harmless action and updating a timestamp metric.

证据

Alertmanager pushes it an alert, not a command.

来源

The gate answers with one of nine verdicts. One means "act".

来源

The gate re‑checks that proof for up to 120 seconds after the action.

来源

The canary alert goes into the chain itself, which runs a harmless canary-touch action that refreshes the timestamp.

来源