48 Hours of Free-Model CI Triage: What I Tried, What Broke, and What I'd Repeat
Описание и применение
Маленькая команда использовала бесплатный LLM для анализа CI‑провалов. Сначала возникли проблемы с лимитами и «галлюцинациями». Добавив пред‑фильтр и заставив модель выдавать структурированный JSON, они сократили вызовы модели на 66 % и повысили надёжность.
Задача
Small teams struggle to triage CI failures efficiently; relying on free LLMs can be costly and unreliable.
Результат
Pre-filtering cut model calls by two thirds, reduced rate-limit errors, and improved trust in model outputs. Structured JSON output enabled automated actions, but confidence scores still required careful handling.
Рабочий процесс
- 1. Connect bot to CI job failures
- 2. Send raw logs to free LLM (naive approach)
- 3. Observe rate limits and hallucinations
- 4. Add deterministic pre-filter to skip obvious failures
- 5. Only ambiguous failures reach the model
- 6. Force model to return JSON with summary, confidence, and action
- 7. Validate JSON with schema before use
- 8. Adjust confidence handling to avoid overconfidence
Доказательства
Three lessons landed in the first morning, and they all pointed in the same direction: stop treating the model like a free-text oracle.
The fix was embarrassingly simple, and it came down to one rule: do not call the model unless the failure is actually ambiguous.
The next change was to force the model into a JSON object with a schema, so the bot could act on the result instead of printing it.
- Дата записи: 2026-09-06T02:51:37.551Z