48 Hours of Free-Model CI Triage: What I Tried, What Broke, and What I'd Repeat
说明与用途
小团队使用免费 LLM 进行 CI 故障 triage。最初的无差别调用导致速率限制和幻觉。通过添加预过滤器并强制模型输出结构化 JSON,减少了 66% 的调用并提高了可靠性。
问题
Small teams struggle to triage CI failures efficiently; relying on free LLMs can be costly and unreliable.
结果
Pre-filtering cut model calls by two thirds, reduced rate-limit errors, and improved trust in model outputs. Structured JSON output enabled automated actions, but confidence scores still required careful handling.
工作流
- 1. Connect bot to CI job failures
- 2. Send raw logs to free LLM (naive approach)
- 3. Observe rate limits and hallucinations
- 4. Add deterministic pre-filter to skip obvious failures
- 5. Only ambiguous failures reach the model
- 6. Force model to return JSON with summary, confidence, and action
- 7. Validate JSON with schema before use
- 8. Adjust confidence handling to avoid overconfidence
证据
Three lessons landed in the first morning, and they all pointed in the same direction: stop treating the model like a free-text oracle.
The fix was embarrassingly simple, and it came down to one rule: do not call the model unless the failure is actually ambiguous.
The next change was to force the model into a JSON object with a schema, so the bot could act on the result instead of printing it.
- 记录日期: 2026-09-06T02:51:37.551Z