I Counted the Turns That Didn't Need a Model

Description and use

Optimizing an agent loop by identifying and eliminating unnecessary model calls, replacing them with simple rules and tools.

Problem

Determining which agent loop turns actually require a language model versus a closed‑set rule or existing tool to avoid unnecessary model calls.

Outcome

The harness revealed that many turns could be handled by simple rules or existing tools, reducing unnecessary model calls and improving response speed.

Workflow

• 1. Receive ticket JSON. 2. Apply gold_policy.gold_plan to decide a closed‑set action plan. 3. Run the agent loop, recording each turn’s action, tool usage, and model usage. 4. Use score_loop.score_transcript to compare the transcript against the gold plan. 5. Classify turns as over‑model, under‑tool, or loop‑overrun. 6. Aggregate scores to evaluate the loop’s efficiency.

Evidence

I built a harness that asks a ruder question than "did the agent finish?"

Source

The loop under test is allowed to look like an agent. It can call tools. It can write a plan.

Source