Regression-testing an agent whose output is never the same twice

说明与用途

冻结代理轨迹并对行为（工具调用、约束、拒绝、评判输出）进行断言的回归套件，使开发者能够在提示或模型更改时检测到隐藏的行为变化。

问题

When a prompt or model changes, it is hard to know what behaviour of an LLM agent has broken because diffing free text is ineffective.

结果

The regression suite detects behavioural changes that would otherwise be invisible, enabling developers to catch regressions quickly and maintain agent reliability.

工作流

• 1. Create a cases.jsonl file with frozen input traces and desired assertions.
• 2. Write assertions.ts to check behaviour (sequence, constraint, refusal, judged).
• 3. Run the test runner after each prompt or model change to compare before and after traces.
• 4. Review movement reports to detect regressions or behavioural shifts.
• 5. Update or add cases as needed to keep the suite relevant.

证据

"Run the suite twice per change: once before, once after. What you report is not pass/fail but movement…"

来源

"Keep the case set small and adversarial. Twenty cases you chose because each one broke something once beats two hundred generated to look thorough…"

来源