I Counted the Turns That Didn't Need a Model

说明与用途

通过识别并消除不必要的模型调用，将代理循环中的简单规则和工具替代，从而优化代理循环。

问题

Determining which agent loop turns actually require a language model versus a closed‑set rule or existing tool to avoid unnecessary model calls.

结果

The harness revealed that many turns could be handled by simple rules or existing tools, reducing unnecessary model calls and improving response speed.

工作流

• 1. Receive ticket JSON. 2. Apply gold_policy.gold_plan to decide a closed‑set action plan. 3. Run the agent loop, recording each turn’s action, tool usage, and model usage. 4. Use score_loop.score_transcript to compare the transcript against the gold plan. 5. Classify turns as over‑model, under‑tool, or loop‑overrun. 6. Aggregate scores to evaluate the loop’s efficiency.

证据

I built a harness that asks a ruder question than "did the agent finish?"

来源

The loop under test is allowed to look like an agent. It can call tools. It can write a plan.

来源