Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

I Let a Free Model Grade Failing Tests for 48 Hours. The Labels Were the Flaky Part

Description and use

Using a free AI model and an aggregator, the author distinguished real failures from flaky ones, reducing false positives while keeping human oversight.

Problem

Determining whether a failing test is a real regression or flaky when the AI model used for triage is itself nondeterministic.

Outcome

By aggregating multiple AI labels and applying a decision table, the team reduced false positives from flaky tests and avoided unnecessary human paging, turning noisy AI outputs into a reliable triage channel.

Workflow

  • 1. Capture test failure details (name, error, stack trace, stdout, diff) and pack into JSON.
  • 2. Send JSON to free AI model with prompt: explain cause in two sentences, then label REAL/FLAKY/UNKNOWN.
  • 3. Store raw AI response and label.
  • 4. Run the job again every four hours to gather additional labels.
  • 5. Aggregate labels: if the same label appears at least twice, accept it; otherwise return NEEDS_RERUN or UNKNOWN.
  • 6. Apply decision table: REAL → page human; FLAKY → quarantine test; NEEDS_RERUN → run once more; UNKNOWN → log and continue.
  • 7. Separate infrastructure failures from test failures before labeling.
  • 8. Review raw logs to audit decisions.

Evidence

The labels were the flakiest part of the system, and the fix was not a better model.

Source

I used MonkeyCode's free model access and free server option because I wanted to see if a zero-budget stack could handle triage without wearing me out.

Source

  • Record date: 2026-09-05T17:09:52.174Z