A 25‑verifier panel measured an effective size of 1.00
说明与用途
一个由25个程序化测试或验收器组成的验证面板,其有效规模仅为1.00,证明增加审阅者并不保证独立证据。
问题
Adding more reviewers to a gate does not guarantee independent evidence; reviewer count is not independent evidence count.
结果
A 25‑verifier panel of programmatic test oracles yielded an effective size of 1.00, indicating that nominal reviewer counts can overstate independent evidence by up to 25×.
工作流
- 1. Define a bounded Work Unit with security bounds.
- 2. Deploy replaceable workers to generate candidate solutions.
- 3. Collect verifier verdicts from a panel of 25 programmatic test oracles.
- 4. Run
idkmesh gate-auditon the collected verdicts to compute panel statistics. - 5. Analyze effective vote size and error correlation to assess panel independence.
证据
The panel was worth no more than one of its members. Twenty‑five nominal votes, an effective size of 1.00, and the standard correlation correction still overstated it by 1.66x.
Experiment E017 built a panel where every verifier is a program, not a model: a partial test oracle that draws inputs from one named region of a problem's input domain and accepts a candidate only if it matches a reference implementation on all of them.
- 记录日期: 2026-09-22T08:00:47.749Z