My LLM Regression Harness Failed 14 Times in 48 Hours. Here’s the Triage I’d Repeat

Описание и применение

Регрессионный harness для LLM часто выдаёт ложные срабатывания из‑за инфраструктурных и логических ошибок. Решения включают разогрев сервера, изоляцию тестов, отдельные таймауты, оценку по рубрике и скрипт классификации ошибок.

Задача

Regression harnesses can misreport failures, leading to wasted effort and mistrust in model quality.

Результат

Only two of the 14 failures were true model regressions; the rest were due to infrastructure, harness design, or rate limits.

Рабочий процесс

• 1. Warm‑up the server with a tiny request before the real run
• 2. Run each test case in isolation with a fresh client
• 3. Separate infrastructure timeouts from model timeouts
• 4. Grade outputs using a rubric instead of exact string matches
• 5. Log status codes separately from outputs
• 6. Add exponential backoff and retry for 429 responses
• 7. Run a triage script to classify failures by log fingerprint
• 8. Count categories to identify flaky components

Доказательства

Fourteen failures came back over 48 hours. How many were the model's fault? Two. The other twelve were my harness, my fixture, or the server's cold shoulder.

Источник

Exact‑match caught wording changes. The model answered "refund policy" where the baseline said "refunds", and my string comparison called it a regression.

Источник

Rate limits masqueraded as bad answers. I fanned out ten parallel calls to speed things up, and the server pushed back with 429s.

Источник