The A/B that measured nothing: three ways my agent experiment was invalid

Описание и применение

Эксперимент по проверке возврата отформатированной строки в MCP провалился из‑за политики шлюза, форматирования результата и противоречивых инструкций. Удаление устаревшего параграфа из инструкций агента исправило проблему.

Задача

The experiment to test whether an MCP tool should return a rendered sentence or raw SI numbers failed three times due to tool-policy gaps, gateway formatting, and contradictory instructions.

Результат

After eliminating the contradictory instruction, the agent consistently quoted the pre-rendered display string, enabling accurate measurement of the A/B test.

Рабочий процесс

• 1. Identify that the agent ignored the display string rule.
• 2. Diagnose three failure points: gateway tool-policy gap, gateway result formatter dropping content, and contradictory workspace instruction.
• 3. Attempt prompt enforcement – failed.
• 4. Attempt structuredContent approach – gateway discarded content, failed.
• 5. Remove the stale paragraph in the workspace instructions that contradicted the persona rule.
• 6. Rerun the A/B test – measurements now reflect correct behavior.

Доказательства

What actually fixed the behaviour was deleting a stale paragraph in the workspace instructions that told the model to do the opposite.

Источник

The gateway renders structuredContent and discards the tool's text content blocks.

Источник