The Alert Storm Got Quieter After I Added a Free Model. Then the Silence Nearly Cost Me the Server.
说明与用途
使用免费 LLM 进行警报分流降低了噪音,但暴露了误判关键警报的风险。
问题
Frequent false alerts from a free server erode trust and cause unnecessary noise, making it hard to distinguish real incidents from noise.
结果
81% accuracy overall; however, critical alerts were sometimes misclassified as ignore, exposing risk of silent failures. The experiment highlighted the importance of robust prompts, error handling, and logging.
工作流
- 1. Point incident webhooks to Python HTTP server.
- 2. Parse alert JSON and store raw payload in SQLite.
- 3. Call MonkeyCode free model with explicit prompt to label alert.
- 4. Parse model JSON response, fallback to watch on errors.
- 5. Store model label and reason next to raw payload.
- 6. Forward critical alerts to notification webhook, watch alerts to daily digest, ignore alerts kept in DB.
- 7. After 48h, manually review alerts, add human_label, compute accuracy and confusion matrix.
证据
The final numbers were an 81% accuracy, which sounds decent until you look at where the errors landed. Two of the three critical alerts were marked ignore
The fallback to watch turned out to be the most important line in the whole service
- 记录日期: 2026-09-05T18:15:10.058Z