Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

48 Hours of Free-Model CI Triage: What I Tried, What Broke, and What I'd Repeat

说明与用途

小团队使用免费 LLM 进行 CI 故障 triage。最初的无差别调用导致速率限制和幻觉。通过添加预过滤器并强制模型输出结构化 JSON,减少了 66% 的调用并提高了可靠性。

问题

Small teams struggle to triage CI failures efficiently; relying on free LLMs can be costly and unreliable.

结果

Pre-filtering cut model calls by two thirds, reduced rate-limit errors, and improved trust in model outputs. Structured JSON output enabled automated actions, but confidence scores still required careful handling.

工作流

  • 1. Connect bot to CI job failures
  • 2. Send raw logs to free LLM (naive approach)
  • 3. Observe rate limits and hallucinations
  • 4. Add deterministic pre-filter to skip obvious failures
  • 5. Only ambiguous failures reach the model
  • 6. Force model to return JSON with summary, confidence, and action
  • 7. Validate JSON with schema before use
  • 8. Adjust confidence handling to avoid overconfidence

证据

Three lessons landed in the first morning, and they all pointed in the same direction: stop treating the model like a free-text oracle.

来源

The fix was embarrassingly simple, and it came down to one rule: do not call the model unless the failure is actually ambiguous.

来源

The next change was to force the model into a JSON object with a schema, so the bot could act on the result instead of printing it.

来源

  • 记录日期: 2026-09-06T02:51:37.551Z