Free Models Are Not Free: 10 CI Pipeline Failures After Adding AI

Description and use

Integrating a free AI model into a CI pipeline caused 10 failures. Fixes include truncating diffs, constraining hallucinations, cleaning JSON, adding retries, limiting concurrency, shortening prompts, caching, error logging, secret redaction, and token monitoring.

Problem

Integrating a free AI model into a CI pipeline for automated code review and commit message generation caused multiple failures due to timeouts, hallucinations, unstable JSON, lack of retries, rate limits, long prompts, duplicate calls, non‑200 responses, sensitive data exposure, and missing usage monitoring.

Outcome

The pipeline became stable and token usage predictable, enabling automated PR review with a free AI model.

Workflow

• 1. Truncate large diffs to 12k characters and append "...[truncated]...".
• 2. Add system prompt instruction "only analyze visible content" to prevent hallucinations.
• 3. Clean model output to strip markdown fences before JSON parsing.
• 4. Wrap API calls in a retry function with exponential backoff and skip after 3 failures.
• 5. Use a semaphore to limit concurrent API calls to 2 to avoid 429 errors.
• 6. Shorten the system prompt to <500 characters to conserve tokens.
• 7. Cache responses keyed by SHA‑256 of the diff to avoid duplicate calls.
• 8. Log non‑200 status codes, response bodies, and latency for debugging.
• 9. Redact API keys, secrets, and tokens from diffs before sending to the model.
• 10. Record prompt and completion token counts to a CSV for usage monitoring.

Evidence

第一个坑出现在第一周。一个重构PR，diff有4万字符。模型调用直接超时。

Source

截断之后，模型开始编造。它引用了一个不存在的文件路径，还煞有介事地分析了那个文件的变更。

Source

模型偶尔在JSON输出外面包一层markdown代码块标记。解析器直接崩。

Source