Free Models Are Not Free: 10 CI Pipeline Failures After Adding AI

说明与用途

将免费 AI 模型集成到 CI 流水线导致 10 次失败。解决方案包括截断 diff、限制幻觉、清理 JSON、添加重试、限制并发、缩短提示词、缓存、错误日志、敏感信息脱敏和 token 监控。

问题

Integrating a free AI model into a CI pipeline for automated code review and commit message generation caused multiple failures due to timeouts, hallucinations, unstable JSON, lack of retries, rate limits, long prompts, duplicate calls, non‑200 responses, sensitive data exposure, and missing usage monitoring.

结果

The pipeline became stable and token usage predictable, enabling automated PR review with a free AI model.

工作流

• 1. Truncate large diffs to 12k characters and append "...[truncated]...".
• 2. Add system prompt instruction "only analyze visible content" to prevent hallucinations.
• 3. Clean model output to strip markdown fences before JSON parsing.
• 4. Wrap API calls in a retry function with exponential backoff and skip after 3 failures.
• 5. Use a semaphore to limit concurrent API calls to 2 to avoid 429 errors.
• 6. Shorten the system prompt to <500 characters to conserve tokens.
• 7. Cache responses keyed by SHA‑256 of the diff to avoid duplicate calls.
• 8. Log non‑200 status codes, response bodies, and latency for debugging.
• 9. Redact API keys, secrets, and tokens from diffs before sending to the model.
• 10. Record prompt and completion token counts to a CSV for usage monitoring.

证据

第一个坑出现在第一周。一个重构PR，diff有4万字符。模型调用直接超时。

来源

截断之后，模型开始编造。它引用了一个不存在的文件路径，还煞有介事地分析了那个文件的变更。

来源

模型偶尔在JSON输出外面包一层markdown代码块标记。解析器直接崩。

来源