The A/B that measured nothing: three ways my agent experiment was invalid

说明与用途

A/B测试因网关策略、结果格式化和相互矛盾的指令而失败。删除工作区中的陈旧指令解决了问题。

问题

The experiment to test whether an MCP tool should return a rendered sentence or raw SI numbers failed three times due to tool-policy gaps, gateway formatting, and contradictory instructions.

结果

After eliminating the contradictory instruction, the agent consistently quoted the pre-rendered display string, enabling accurate measurement of the A/B test.

工作流

• 1. Identify that the agent ignored the display string rule.
• 2. Diagnose three failure points: gateway tool-policy gap, gateway result formatter dropping content, and contradictory workspace instruction.
• 3. Attempt prompt enforcement – failed.
• 4. Attempt structuredContent approach – gateway discarded content, failed.
• 5. Remove the stale paragraph in the workspace instructions that contradicted the persona rule.
• 6. Rerun the A/B test – measurements now reflect correct behavior.

证据

What actually fixed the behaviour was deleting a stale paragraph in the workspace instructions that told the model to do the opposite.

来源

The gateway renders structuredContent and discards the tool's text content blocks.

来源