One Prometheus for unRaid, a Synology NAS, and a VPS: What I Got Wrong

Description and use

Three machines (unRaid, Synology NAS, VPS) were unified into a single Prometheus/Grafana monitoring panel. Issues fixed: duplicate filesystems, missing CPU frequency on the guest, incorrect cAdvisor metrics, and wrong host names. Result: accurate metrics and timely alerts.

Problem

Three separate machines (unRaid, Synology NAS, VPS) were monitored independently, leading to inaccurate metrics, misleading alerts, and hidden resource issues.

Outcome

Accurate single-screen monitoring of 3 hosts with 7 scrape targets, 16 alert rules, and clear metrics: 47 CPU cores, 158 GHz current, 142 GB RAM, 64 TB storage, 155 running containers.

Workflow

• Deploy node_exporter and cAdvisor on each host
• Configure node_exporter to expose host metrics on :9100
• Configure cAdvisor to expose container metrics on :8080, :8081, :8082
• Add filter to cAdvisor metrics: node_memory_working_set_bytes{job=~"cadvisor.*", name!=""}
• Create Prometheus query to sum node_filesystem_size_bytes with mountpoint and fstype filters
• Add textfile collector script on VPS to expose CPU MHz as node_cpu_mhz_current_hz
• Configure node_exporter to use hostname from compose file
• Build Grafana dashboards for host and container metrics
• Create alert rules for storage, CPU, memory, and container counts
• Test and iterate until dashboards and alerts reflect true state

Evidence

node_exporter sees the host: CPU, memory, network, disks, filesystems, temperatures. It is cgroup-blind: it cannot tell you which container is eating your RAM.

Source

The fix is one filter, but you have to know to write it: # container metrics: anything with a name, and only that container_memory_working_set_bytes{job=~"cadvisor.*", name!=""}

Source

A naive sum(node_filesystem_size_bytes) therefore reported capacity for disks I don't have.

Source

The VPS reported nothing at all — it's a KVM guest, and a guest has no /sys/devices/system/cpu/cpu0/cpufreq.

Source

The host dropdown in the dashboard offered 9f9afcccc962 as a machine. That's a container ID: node_exporter's uname collector reports the process's UTS namespace...

Source