Our agent said "done" on 15% of tasks while the provider was failing

说明与用途

Altair 由于提供商错误错误地将任务标记为完成；在添加答案验证和重试后，错误完成率降至零。

问题

An AI agent reported task completion for 15% of tasks, but the provider returned errors or empty streams, causing false "done" signals.

结果

False "done" rate dropped from 15% to near 0% after adding error detection and retry logic.

工作流

• Run 46 tasks with Altair
• Log provider responses via proxy
• Detect false "done" using judge_turn
• Retry or report error
• Re‑run tasks with updated logic

证据

7 weren't, and in all seven the provider broke: 6 times — the gateway's error text instead of an answer. Once — an empty stream.

来源

We now check every answer without a tool call. It's not an answer if: ...

来源