My LLM Regression Harness Failed 14 Times in 48 Hours. Here’s the Triage I’d Repeat

说明与用途

LLM 回归 harness 经常因基础设施和设计问题产生误报。解决方案包括热身调用、每个测试隔离、单独超时、基于评分表的评估以及日志指纹分类脚本。

问题

Regression harnesses can misreport failures, leading to wasted effort and mistrust in model quality.

结果

Only two of the 14 failures were true model regressions; the rest were due to infrastructure, harness design, or rate limits.

工作流

• 1. Warm‑up the server with a tiny request before the real run
• 2. Run each test case in isolation with a fresh client
• 3. Separate infrastructure timeouts from model timeouts
• 4. Grade outputs using a rubric instead of exact string matches
• 5. Log status codes separately from outputs
• 6. Add exponential backoff and retry for 429 responses
• 7. Run a triage script to classify failures by log fingerprint
• 8. Count categories to identify flaky components

证据

Fourteen failures came back over 48 hours. How many were the model's fault? Two. The other twelve were my harness, my fixture, or the server's cold shoulder.

来源

Exact‑match caught wording changes. The model answered "refund policy" where the baseline said "refunds", and my string comparison called it a regression.

来源

Rate limits masqueraded as bad answers. I fanned out ten parallel calls to speed things up, and the server pushed back with 429s.

来源