Tracing one alert through Alertmanager

Описание и применение

Автор исследовал, как Alertmanager обрабатывает отдельный сигнал, обнаружив, что уведомления проходят через группу таймеров и проверку дублирования после группировки, а не в линейном потоке.

Задача

Understanding how Alertmanager separates signal from noise when many alerts arrive at once.

Результат

The trace confirmed that alerts are stored only in the Alert Provider, that the Dispatcher and Inhibitor receive alerts independently, that the group timer separates insertion from pipeline execution, and that deduplication occurs after the timer flush, preventing duplicate notifications.

Рабочий процесс

• 1. Alert ingestion via HTTP API writes to Alert Provider.
• 2. Silence rules are created via separate POST to Silences.Set and read later.
• 3. Inhibitor (background process) receives alert to track symptom relationships.
• 4. Dispatcher receives alert, matches route, inserts into group, starts group timer.
• 5. After timer flushes, notification processing begins: cluster settle, suppression checks, silence rule application, route selection, integration branch selection.
• 6. Deduplication decision is made before sending notification.
• 7. If allowed, webhook integration is called and notification is logged.
• 8. Subsequent group flushes trigger same checks; deduplication blocks repeat notifications.

Доказательства

The summary said the alert-ingestion request writes only to the Alert Provider.

Источник

The Inhibitor is a background process that tracks whether one alert is only a symptom of another.

Источник

The group timer separates insertion from pipeline execution.

Источник

Deduplication decision is made before sending notification.

Источник