Your LLM vendor can change its mind overnight. Here’s how we started catching it.

Описание и применение

Автоматизированные регрессионные тесты обнаружили скрытое изменение ответа модели Jev, позволяя команде быстро реагировать и поддерживать надёжность приложения.

Задача

LLM APIs can silently change outputs after a model update, causing downstream logic to break without any error or changelog.

Результат

Automated regression tests detected a silent change in Jev’s output, allowing the team to respond immediately and maintain application reliability.

Рабочий процесс

• Write a JSON test case defining the scenario, question, expected answer, and tolerance.
• Commit the test case to the repository as a baseline.
• Run jev-watch against the test suite after each model update, on a schedule, or in CI.
• Review any FAIL output to identify drift in specific fields.
• Fix application logic or update test expectations as needed.
• Use exit codes to fail CI when tests fail.
• Optionally run against OpenRouter to avoid holding a TypeSafe key.

Доказательства

We only caught it by comparing live output against a set of answers we knew were correct.

Источник

$ jev-watch examples
PASS CRAToolkit refund eligibility
PASS CRAToolkit sales lead qualification
PASS CRAToolkit angry customer detection
PASS CRAToolkit support routing

Источник