My self-healing homelab saved 17 seconds. Here’s why I kept it

Описание и применение

Автоматизированная цепочка реагирования в домашнем лабораторном окружении сократила время простоя на 17 секунд, предотвращая цепные перезапуски и обеспечивая аудит действий.

Задача

A power cut caused the reverse proxy to fail, delaying alerts and leaving sites inaccessible until the issue was discovered hours later.

Результат

The automated chain restored services 17 seconds faster than a human responder, prevented cascading restarts, and logged all actions for audit.

Рабочий процесс

• 1. Prometheus detects service health via probes.
• 2. Alertmanager forwards alerts to n8n via webhook.
• 3. n8n drops stale alerts and forwards the current alert to the policy gate.
• 4. The policy gate evaluates the alert against a closed catalogue of actions and returns a verdict (act or refuse).
• 5. If act, the gate sends the action to the executor (rb-exec) which runs a single command (e.g., docker restart).
• 6. The gate re‑checks Prometheus for proof of recovery up to 120 s after the action.
• 7. If recovery fails, the action is logged as a failure and the user is paged.
• 8. Guardrails (circuit breaker, no‑op counting, failure thresholds) prevent runaway remediation.
• 9. A canary alert periodically verifies the chain’s health by running a harmless action and updating a timestamp metric.

Доказательства

Alertmanager pushes it an alert, not a command.

Источник

The gate answers with one of nine verdicts. One means "act".

Источник

The gate re‑checks that proof for up to 120 seconds after the action.

Источник

The canary alert goes into the chain itself, which runs a harmless canary-touch action that refreshes the timestamp.

Источник