The two-lane eval: model score vs product score for an MCP agent

Description and use

<p><strong>TL;DR</strong></p> <ul> <li>A <strong>two-lane eval</strong> runs the same cases against the model alone (model lane) and through the whole product (product lane), and keeps the scores separate.</li> <li>In August 2026 the same Qwen3-4B agent scored <strong>42/70 in the model lane</strong> (failed the ship gate) and <strong>46/70 in the product lane</strong> (passed it).</li> <li>One guard, the <strong>stall rule</strong> (bounce a turn that neither calls a tool nor asks the user), earned the entire gain. The argument repair I was proud of measured zero.</li> </ul> <p>I'm building a

Problem

<p><strong>TL;DR</strong></p> <ul> <li>A <strong>two-lane eval</strong> runs the same cases against the model alone (model lane) and through the whole product (product lane), and keeps the scores separate.</li> <li>In August 2026 the same Qwen3-4B agent scored <strong>42/70 in the model lane</strong> (failed the ship gate) and <strong>46/70 in the product lane</strong> (passed it).</li> <li>One guard, the <strong>stall rule</strong> (bounce a turn that neither calls a tool nor asks the user), earned the entire gain. The argument repair I was proud of measured zero.</li> </ul> <p>I'm building a

Outcome

<p><strong>TL;DR</strong></p> <ul> <li>A <strong>two-lane eval</strong> runs the same cases against the model alone (model lane) and through the whole product (product lane), and keeps the scores separate.</li> <li>In August 2026 the same Qwen3-4B agent scored <strong>42/70 in the model lane</strong> (failed the ship gate) and <strong>46/70 in the product lane</strong> (passed it).</li> <li>One guard, the <strong>stall rule</strong> (bounce a turn that neither calls a tool nor asks the user), earned the entire gain. The argument repair I was proud of measured zero.</li> </ul> <p>I'm building a

Evidence

The two-lane eval: model score vs product score for an MCP agent

Source