Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

48 Hours of Free-Model CI Triage: What I Tried, What Broke, and What I'd Repeat

Описание и применение

Маленькая команда использовала бесплатный LLM для анализа CI‑провалов. Сначала возникли проблемы с лимитами и «галлюцинациями». Добавив пред‑фильтр и заставив модель выдавать структурированный JSON, они сократили вызовы модели на 66 % и повысили надёжность.

Задача

Small teams struggle to triage CI failures efficiently; relying on free LLMs can be costly and unreliable.

Результат

Pre-filtering cut model calls by two thirds, reduced rate-limit errors, and improved trust in model outputs. Structured JSON output enabled automated actions, but confidence scores still required careful handling.

Рабочий процесс

  • 1. Connect bot to CI job failures
  • 2. Send raw logs to free LLM (naive approach)
  • 3. Observe rate limits and hallucinations
  • 4. Add deterministic pre-filter to skip obvious failures
  • 5. Only ambiguous failures reach the model
  • 6. Force model to return JSON with summary, confidence, and action
  • 7. Validate JSON with schema before use
  • 8. Adjust confidence handling to avoid overconfidence

Доказательства

Three lessons landed in the first morning, and they all pointed in the same direction: stop treating the model like a free-text oracle.

Источник

The fix was embarrassingly simple, and it came down to one rule: do not call the model unless the failure is actually ambiguous.

Источник

The next change was to force the model into a JSON object with a schema, so the bot could act on the result instead of printing it.

Источник

  • Дата записи: 2026-09-06T02:51:37.551Z