Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

I Let a Free Model Grade Failing Tests for 48 Hours. The Labels Were the Flaky Part

说明与用途

通过使用免费 AI 模型和聚合器,作者区分了真实错误和偶发错误,减少了误报并保持了人工监督。

问题

Determining whether a failing test is a real regression or flaky when the AI model used for triage is itself nondeterministic.

结果

By aggregating multiple AI labels and applying a decision table, the team reduced false positives from flaky tests and avoided unnecessary human paging, turning noisy AI outputs into a reliable triage channel.

工作流

  • 1. Capture test failure details (name, error, stack trace, stdout, diff) and pack into JSON.
  • 2. Send JSON to free AI model with prompt: explain cause in two sentences, then label REAL/FLAKY/UNKNOWN.
  • 3. Store raw AI response and label.
  • 4. Run the job again every four hours to gather additional labels.
  • 5. Aggregate labels: if the same label appears at least twice, accept it; otherwise return NEEDS_RERUN or UNKNOWN.
  • 6. Apply decision table: REAL → page human; FLAKY → quarantine test; NEEDS_RERUN → run once more; UNKNOWN → log and continue.
  • 7. Separate infrastructure failures from test failures before labeling.
  • 8. Review raw logs to audit decisions.

证据

The labels were the flakiest part of the system, and the fix was not a better model.

来源

I used MonkeyCode's free model access and free server option because I wanted to see if a zero-budget stack could handle triage without wearing me out.

来源

  • 记录日期: 2026-09-05T17:09:52.174Z