I Let a Free Model Grade Failing Tests for 48 Hours. The Labels Were the Flaky Part
说明与用途
通过使用免费 AI 模型和聚合器,作者区分了真实错误和偶发错误,减少了误报并保持了人工监督。
问题
Determining whether a failing test is a real regression or flaky when the AI model used for triage is itself nondeterministic.
结果
By aggregating multiple AI labels and applying a decision table, the team reduced false positives from flaky tests and avoided unnecessary human paging, turning noisy AI outputs into a reliable triage channel.
工作流
- 1. Capture test failure details (name, error, stack trace, stdout, diff) and pack into JSON.
- 2. Send JSON to free AI model with prompt: explain cause in two sentences, then label REAL/FLAKY/UNKNOWN.
- 3. Store raw AI response and label.
- 4. Run the job again every four hours to gather additional labels.
- 5. Aggregate labels: if the same label appears at least twice, accept it; otherwise return NEEDS_RERUN or UNKNOWN.
- 6. Apply decision table: REAL → page human; FLAKY → quarantine test; NEEDS_RERUN → run once more; UNKNOWN → log and continue.
- 7. Separate infrastructure failures from test failures before labeling.
- 8. Review raw logs to audit decisions.
证据
The labels were the flakiest part of the system, and the fix was not a better model.
I used MonkeyCode's free model access and free server option because I wanted to see if a zero-budget stack could handle triage without wearing me out.
- 记录日期: 2026-09-05T17:09:52.174Z