The A/B that measured nothing: three ways my agent experiment was invalid

Description and use

The A/B test to verify MCP tool output failed due to gateway policy gaps, result formatting, and contradictory instructions. Removing a stale instruction from the agent’s workspace fixed the issue.

Problem

The experiment to test whether an MCP tool should return a rendered sentence or raw SI numbers failed three times due to tool-policy gaps, gateway formatting, and contradictory instructions.

Outcome

After eliminating the contradictory instruction, the agent consistently quoted the pre-rendered display string, enabling accurate measurement of the A/B test.

Workflow

• 1. Identify that the agent ignored the display string rule.
• 2. Diagnose three failure points: gateway tool-policy gap, gateway result formatter dropping content, and contradictory workspace instruction.
• 3. Attempt prompt enforcement – failed.
• 4. Attempt structuredContent approach – gateway discarded content, failed.
• 5. Remove the stale paragraph in the workspace instructions that contradicted the persona rule.
• 6. Rerun the A/B test – measurements now reflect correct behavior.

Evidence

What actually fixed the behaviour was deleting a stale paragraph in the workspace instructions that told the model to do the opposite.

Source

The gateway renders structuredContent and discards the tool's text content blocks.

Source