Pin Behaviors Across Model Swaps

Description and use

Validating LLM behavior with golden files and independent graders catches regressions invisible to surface‑form snapshot tests.

Problem

Silent LLM regressions survive snapshot tests because the payload still parses and the last message still looks fluent, making surface‑form mismatches invisible.

Outcome

The harness detects silent regressions such as missing identifiers, tool misuse, or system text leakage that surface‑form tests miss, enabling safer model swaps.

Workflow

• 1. Create a JSON golden case file with id, frozen user turn, system prompt hash, allowed tools, and required behaviors.
• 2. Run the target model against the user turn and capture the full trace (final_text, tool_calls).
• 3. Apply grader functions (e.g., json_object, no_system_leak, max_tool_rounds) to the trace to evaluate each behavior.
• 4. Aggregate results into a deterministic report; failures are flagged automatically.
• 5. If a regression is detected, review the trace, adjust the model or prompt, and re‑run the harness until all behaviors pass.

Evidence

Silent LLM regressions survive snapshot tests because the payload still parses and the last message still looks fluent.

Source

A harness that versions golden behaviors and scores them with independent graders catches those failures before a model swap ships.

Source