My LLM Regression Harness Failed 14 Times in 48 Hours. Here’s the Triage I’d Repeat

Description and use

An LLM regression harness frequently reports false failures due to infrastructure and design issues. Remedies include warm‑up calls, per‑case isolation, separate timeouts, rubric grading, and a log‑fingerprinting triage script.

Problem

Regression harnesses can misreport failures, leading to wasted effort and mistrust in model quality.

Outcome

Only two of the 14 failures were true model regressions; the rest were due to infrastructure, harness design, or rate limits.

Workflow

• 1. Warm‑up the server with a tiny request before the real run
• 2. Run each test case in isolation with a fresh client
• 3. Separate infrastructure timeouts from model timeouts
• 4. Grade outputs using a rubric instead of exact string matches
• 5. Log status codes separately from outputs
• 6. Add exponential backoff and retry for 429 responses
• 7. Run a triage script to classify failures by log fingerprint
• 8. Count categories to identify flaky components

Evidence

Fourteen failures came back over 48 hours. How many were the model's fault? Two. The other twelve were my harness, my fixture, or the server's cold shoulder.

Source

Exact‑match caught wording changes. The model answered "refund policy" where the baseline said "refunds", and my string comparison called it a regression.

Source

Rate limits masqueraded as bad answers. I fanned out ten parallel calls to speed things up, and the server pushed back with 429s.

Source