My self-healing homelab saved 17 seconds. Here’s why I kept it

Description and use

An automated remediation chain in a home lab reduced downtime by 17 seconds, prevented cascading restarts, and logged all actions for audit.

A power cut caused the reverse proxy to fail, delaying alerts and leaving sites inaccessible until the issue was discovered hours later.

The automated chain restored services 17 seconds faster than a human responder, prevented cascading restarts, and logged all actions for audit.

1. Prometheus detects service health via probes.
2. Alertmanager forwards alerts to n8n via webhook.
3. n8n drops stale alerts and forwards the current alert to the policy gate.
4. The policy gate evaluates the alert against a closed catalogue of actions and returns a verdict (act or refuse).
5. If act, the gate sends the action to the executor (rb-exec) which runs a single command (e.g., docker restart).
6. The gate re‑checks Prometheus for proof of recovery up to 120 s after the action.
7. If recovery fails, the action is logged as a failure and the user is paged.
8. Guardrails (circuit breaker, no‑op counting, failure thresholds) prevent runaway remediation.
9. A canary alert periodically verifies the chain’s health by running a harmless action and updating a timestamp metric.

Alertmanager pushes it an alert, not a command.

The gate answers with one of nine verdicts. One means "act".

The gate re‑checks that proof for up to 120 seconds after the action.

The canary alert goes into the chain itself, which runs a harmless canary-touch action that refreshes the timestamp.