Regression-testing an agent whose output is never the same twice

Описание и применение

Набор тестов, фиксирующий трассы действий агента, позволяет обнаруживать регрессии в поведении LLM‑агента при изменении подсказок или моделей, используя утверждения о последовательности вызовов инструментов, ограничениях и отказах.

Задача

When a prompt or model changes, it is hard to know what behaviour of an LLM agent has broken because diffing free text is ineffective.

Результат

The regression suite detects behavioural changes that would otherwise be invisible, enabling developers to catch regressions quickly and maintain agent reliability.

Рабочий процесс

• 1. Create a cases.jsonl file with frozen input traces and desired assertions.
• 2. Write assertions.ts to check behaviour (sequence, constraint, refusal, judged).
• 3. Run the test runner after each prompt or model change to compare before and after traces.
• 4. Review movement reports to detect regressions or behavioural shifts.
• 5. Update or add cases as needed to keep the suite relevant.

Доказательства

"Run the suite twice per change: once before, once after. What you report is not pass/fail but movement…"

Источник

"Keep the case set small and adversarial. Twenty cases you chose because each one broke something once beats two hundred generated to look thorough…"

Источник