The Alert Storm Got Quieter After I Added a Free Model. Then the Silence Nearly Cost Me the Server.
Description and use
Using a free LLM for alert triage reduced noise but exposed risks of misclassifying critical alerts.
Problem
Frequent false alerts from a free server erode trust and cause unnecessary noise, making it hard to distinguish real incidents from noise.
Outcome
81% accuracy overall; however, critical alerts were sometimes misclassified as ignore, exposing risk of silent failures. The experiment highlighted the importance of robust prompts, error handling, and logging.
Workflow
- 1. Point incident webhooks to Python HTTP server.
- 2. Parse alert JSON and store raw payload in SQLite.
- 3. Call MonkeyCode free model with explicit prompt to label alert.
- 4. Parse model JSON response, fallback to watch on errors.
- 5. Store model label and reason next to raw payload.
- 6. Forward critical alerts to notification webhook, watch alerts to daily digest, ignore alerts kept in DB.
- 7. After 48h, manually review alerts, add human_label, compute accuracy and confusion matrix.
Evidence
The final numbers were an 81% accuracy, which sounds decent until you look at where the errors landed. Two of the three critical alerts were marked ignore
The fallback to watch turned out to be the most important line in the whole service
- Record date: 2026-09-05T18:15:10.058Z