Pin Behaviors Across Model Swaps

说明与用途

使用黄金文件和独立评估器验证LLM行为，可捕捉仅靠表面文本快照测试无法发现的回归。

问题

Silent LLM regressions survive snapshot tests because the payload still parses and the last message still looks fluent, making surface‑form mismatches invisible.

结果

The harness detects silent regressions such as missing identifiers, tool misuse, or system text leakage that surface‑form tests miss, enabling safer model swaps.

工作流

• 1. Create a JSON golden case file with id, frozen user turn, system prompt hash, allowed tools, and required behaviors.
• 2. Run the target model against the user turn and capture the full trace (final_text, tool_calls).
• 3. Apply grader functions (e.g., json_object, no_system_leak, max_tool_rounds) to the trace to evaluate each behavior.
• 4. Aggregate results into a deterministic report; failures are flagged automatically.
• 5. If a regression is detected, review the trace, adjust the model or prompt, and re‑run the harness until all behaviors pass.

证据

Silent LLM regressions survive snapshot tests because the payload still parses and the last message still looks fluent.

来源

A harness that versions golden behaviors and scores them with independent graders catches those failures before a model swap ships.

来源