Run a watchdog that doesn't live in the cloud it watches

Description and use

A self‑hosted watchdog on local hardware provides an independent monitoring source that can detect outages even when cloud monitoring services are down, improving incident response.

Problem

Monitoring services often run on the same cloud infrastructure they monitor, causing outages when that infrastructure fails, leaving teams unable to detect or respond to failures.

Outcome

Teams gain an always‑available watchdog that can detect outages even when cloud monitoring services are down, improving incident response time and reliability.

Workflow

• Deploy a single Docker container on local hardware
• Configure the container to check internet uplink against multiple anchors
• Add DNS checks for critical domains
• Set up HTTP assertions for public services (status code, keyword in body)
• Monitor TLS certificate and domain expiry
• Include local infrastructure checks (containers, hypervisor, NAS, temperatures)
• Configure alerts to use existing notification channels

Evidence

When AWS's us-east-1 went down in October, and Cloudflare followed in November, plenty of teams learned that their monitoring lived on the same infrastructure as the things it monitored.

Source

The fix is old-fashioned and cheap: keep one monitor that depends on nothing but a box in your own building.

Source