Tracing one alert through Alertmanager

Description and use

The author traced a single alert in Alertmanager, discovering that alerts are stored only in the Alert Provider, that the Dispatcher and Inhibitor receive alerts independently, and that deduplication occurs after the group timer flush, preventing duplicate notifications.

Problem

Understanding how Alertmanager separates signal from noise when many alerts arrive at once.

Outcome

The trace confirmed that alerts are stored only in the Alert Provider, that the Dispatcher and Inhibitor receive alerts independently, that the group timer separates insertion from pipeline execution, and that deduplication occurs after the timer flush, preventing duplicate notifications.

Workflow

• 1. Alert ingestion via HTTP API writes to Alert Provider.
• 2. Silence rules are created via separate POST to Silences.Set and read later.
• 3. Inhibitor (background process) receives alert to track symptom relationships.
• 4. Dispatcher receives alert, matches route, inserts into group, starts group timer.
• 5. After timer flushes, notification processing begins: cluster settle, suppression checks, silence rule application, route selection, integration branch selection.
• 6. Deduplication decision is made before sending notification.
• 7. If allowed, webhook integration is called and notification is logged.
• 8. Subsequent group flushes trigger same checks; deduplication blocks repeat notifications.

Evidence

The summary said the alert-ingestion request writes only to the Alert Provider.

Source

The Inhibitor is a background process that tracks whether one alert is only a symptom of another.

Source

The group timer separates insertion from pipeline execution.

Source

Deduplication decision is made before sending notification.

Source