My Audit Caught Every Known Bug. Ten of Its Next 18 Flags Were Wrong.

Description and use

An audit checklist based on known defects produced many false positives on new guards, misleading precision metrics. Recalculating precision yielded 44.4% on resolved findings. Tightening criteria reduced findings but did not conclusively prove improvement due to sample changes. A reporting pattern separating accepted, rejected, and unresolved findings prevents denominator mistakes.

Problem

A checklist built from known defects produced many false positives when applied to new guards, leading to misleading precision metrics and unclear reporting.

Outcome

The audit revealed that the checklist’s precision was 44.4% on resolved findings, not 56% as previously claimed. Refining criteria reduced false positives but did not conclusively prove improvement due to sample changes. A reporting pattern that separates accepted, rejected, and unresolved findings prevents denominator mistakes.

Workflow

• 1. Run AI agents to audit guards and record findings and adjudications.
• 2. Reconstruct counts of accepted and rejected findings from audit logs.
• 3. Calculate precision as accepted/(accepted+rejected).
• 4. Identify over‑broad criteria (e.g., empty‑input) and refine contracts.
• 5. Apply tightened criteria to a new guard sample and audit again.
• 6. Use a Python script to report counts and precision, keeping unresolved findings separate.

Evidence

A checklist built from seven known defects caught all seven, which only proves it can describe the failures that shaped it.

Source

For the first audit, 8/18 findings were accepted: 44.4% precision under the recorded labels.

Source

The old wording collapsed those cases. A better contract asks what the program has promised to verify.

Source