Your LLM vendor can change its mind overnight. Here’s how we started catching it.

说明与用途

自动化回归测试检测到 JEV 模型输出的静默变化，使团队能够立即响应并保持应用程序的可靠性。

问题

LLM APIs can silently change outputs after a model update, causing downstream logic to break without any error or changelog.

结果

Automated regression tests detected a silent change in Jev’s output, allowing the team to respond immediately and maintain application reliability.

工作流

• Write a JSON test case defining the scenario, question, expected answer, and tolerance.
• Commit the test case to the repository as a baseline.
• Run jev-watch against the test suite after each model update, on a schedule, or in CI.
• Review any FAIL output to identify drift in specific fields.
• Fix application logic or update test expectations as needed.
• Use exit codes to fail CI when tests fail.
• Optionally run against OpenRouter to avoid holding a TypeSafe key.

证据

We only caught it by comparing live output against a set of answers we knew were correct.

来源

$ jev-watch examples
PASS CRAToolkit refund eligibility
PASS CRAToolkit sales lead qualification
PASS CRAToolkit angry customer detection
PASS CRAToolkit support routing

来源