Your LLM vendor can change its mind overnight. Here’s how we started catching it.

Description and use

Automated regression tests caught a silent change in Jev’s output, enabling the team to respond immediately and maintain application reliability.

Problem

LLM APIs can silently change outputs after a model update, causing downstream logic to break without any error or changelog.

Outcome

Automated regression tests detected a silent change in Jev’s output, allowing the team to respond immediately and maintain application reliability.

Workflow

• Write a JSON test case defining the scenario, question, expected answer, and tolerance.
• Commit the test case to the repository as a baseline.
• Run jev-watch against the test suite after each model update, on a schedule, or in CI.
• Review any FAIL output to identify drift in specific fields.
• Fix application logic or update test expectations as needed.
• Use exit codes to fail CI when tests fail.
• Optionally run against OpenRouter to avoid holding a TypeSafe key.

Evidence

We only caught it by comparing live output against a set of answers we knew were correct.

Source

$ jev-watch examples
PASS CRAToolkit refund eligibility
PASS CRAToolkit sales lead qualification
PASS CRAToolkit angry customer detection
PASS CRAToolkit support routing

Source