Pin the Envelope, Not the Model Name

Description and use

Tripwire using three digests detects silent model swaps in free/trial endpoints, preventing regression in evaluation scores.

Problem

Regression from routing changes in free/trial endpoints can silently alter evaluation scores.

Outcome

Tripwire detects silent model swaps by comparing request, envelope, and stamp digests, preventing unnoticed regression in evaluation scores.

Workflow

• 1. Generate request fingerprint of the prompt and parameters.
• 2. Compute envelope digest of the response text, finish reason, and usage keys.
• 3. Record declared stamp (endpoint identity).
• 4. Store all three digests per run in a JSON line file.
• 5. On subsequent runs, compare request, envelope, and stamp digests to baseline.
• 6. If request digest unchanged but envelope digest differs and stamp unchanged, flag a potential silent model swap.
• 7. Use paired sign test to assess statistical significance of score changes.

Evidence

Every call should produce three independent digests, because each one answers a different question about who changed what.

Source

Write one JSON line per case per run, and never overwrite the file.

Source