A 25‑verifier panel measured an effective size of 1.00

Description and use

A 25‑verifier panel of programmatic test oracles had an effective size of only 1.00, proving that adding reviewers does not guarantee independent evidence.

Problem

Adding more reviewers to a gate does not guarantee independent evidence; reviewer count is not independent evidence count.

Outcome

A 25‑verifier panel of programmatic test oracles yielded an effective size of 1.00, indicating that nominal reviewer counts can overstate independent evidence by up to 25×.

Workflow

• 1. Define a bounded Work Unit with security bounds.
• 2. Deploy replaceable workers to generate candidate solutions.
• 3. Collect verifier verdicts from a panel of 25 programmatic test oracles.
• 4. Run idkmesh gate-audit on the collected verdicts to compute panel statistics.
• 5. Analyze effective vote size and error correlation to assess panel independence.

Evidence

The panel was worth no more than one of its members. Twenty‑five nominal votes, an effective size of 1.00, and the standard correlation correction still overstated it by 1.66x.

Source

Experiment E017 built a panel where every verifier is a program, not a model: a partial test oracle that draws inputs from one named region of a problem's input domain and accepts a candidate only if it matches a reference implementation on all of them.

Source