My Audit Caught Every Known Bug. Ten of Its Next 18 Flags Were Wrong.

说明与用途

基于已知缺陷的检查清单在新守卫上产生了大量误报，导致精度指标误导。重新计算精度后，已解决发现的精度为44.4%。收紧标准后发现数减少，但由于样本变化，无法明确证明改进。引入报告模式，将已接受、已拒绝和未解决的发现分开，避免分母错误。

问题

A checklist built from known defects produced many false positives when applied to new guards, leading to misleading precision metrics and unclear reporting.

结果

The audit revealed that the checklist’s precision was 44.4% on resolved findings, not 56% as previously claimed. Refining criteria reduced false positives but did not conclusively prove improvement due to sample changes. A reporting pattern that separates accepted, rejected, and unresolved findings prevents denominator mistakes.

工作流

• 1. Run AI agents to audit guards and record findings and adjudications.
• 2. Reconstruct counts of accepted and rejected findings from audit logs.
• 3. Calculate precision as accepted/(accepted+rejected).
• 4. Identify over‑broad criteria (e.g., empty‑input) and refine contracts.
• 5. Apply tightened criteria to a new guard sample and audit again.
• 6. Use a Python script to report counts and precision, keeping unresolved findings separate.

证据

A checklist built from seven known defects caught all seven, which only proves it can describe the failures that shaped it.

来源

For the first audit, 8/18 findings were accepted: 44.4% precision under the recorded labels.

来源

The old wording collapsed those cases. A better contract asks what the program has promised to verify.

来源