Tracing one alert through Alertmanager

说明与用途

作者追踪了 Alertmanager 中的单个告警，发现告警仅写入 Alert Provider，Dispatcher 和 Inhibitor 独立接收告警，且去重检查发生在组定时器刷新后，防止重复通知。

问题

Understanding how Alertmanager separates signal from noise when many alerts arrive at once.

结果

The trace confirmed that alerts are stored only in the Alert Provider, that the Dispatcher and Inhibitor receive alerts independently, that the group timer separates insertion from pipeline execution, and that deduplication occurs after the timer flush, preventing duplicate notifications.

工作流

• 1. Alert ingestion via HTTP API writes to Alert Provider.
• 2. Silence rules are created via separate POST to Silences.Set and read later.
• 3. Inhibitor (background process) receives alert to track symptom relationships.
• 4. Dispatcher receives alert, matches route, inserts into group, starts group timer.
• 5. After timer flushes, notification processing begins: cluster settle, suppression checks, silence rule application, route selection, integration branch selection.
• 6. Deduplication decision is made before sending notification.
• 7. If allowed, webhook integration is called and notification is logged.
• 8. Subsequent group flushes trigger same checks; deduplication blocks repeat notifications.

证据

The summary said the alert-ingestion request writes only to the Alert Provider.

来源

The Inhibitor is a background process that tracks whether one alert is only a symptom of another.

来源

The group timer separates insertion from pipeline execution.

来源

Deduplication decision is made before sending notification.

来源