My security tool missed my reverse proxy going down. So warden v3 watches the boxes too

Description and use

Warden v3 added heartbeat checks for containers and nodes, detected a stopped reverse proxy after a NAS OS update, and now auto‑draws network maps and logs SSH activity.

Problem

A security monitoring tool failed to detect that a reverse proxy container had stopped after a NAS OS update, causing a 35‑minute outage.

Outcome

Warden v3 now detects stopped containers, tracks uptime, logs SSH activity, and auto‑draws network topology, reducing missed outages and improving visibility.

Workflow

• 1. Update NAS OS, triggering a reboot. 2. Docker stops containers during shutdown. 3. Containers do not restart due to "restart: unless-stopped" policy. 4. Warden v3 performs heartbeat checks every 5 minutes. 5. Warden detects missing beats for the reverse proxy container. 6. Warden sends a notification about the outage and subsequent recovery.
• 7. Warden logs container uptime and identifies if a reboot occurred.
• 8. Warden records SSH sign‑ins and failed attempts per box.
• 9. Warden generates an auto‑drawn network map from known topology data.

Evidence

Turns out the update had run docker stop on its way down, and Docker treats that as you stopping them on purpose.

Source

Every 5 minutes warden checks every server and container: up, down, stopped, or unknown, and how long it's been up.

Source

If a box misses three beats you get one message. When it's back you get another, and if the uptime is shorter than the outage it tells you it rebooted.

Source

I wanted a network diagram but I didn't want to draw one, because a hand-drawn diagram is wrong the week after you draw it.

Source