My Audit Caught Every Known Bug. Ten of Its Next 18 Flags Were Wrong.

Описание и применение

Проверка чек-листа, основанного на известных дефектах, выявила множество ложных срабатываний при применении к новым гвардам. Пересчёт показателей точности показал 44,4 % точности среди решённых находок, а не 56 %. Пересмотр критериев сократил количество находок, но не доказал улучшения из‑за изменения выборки. Внедрён шаблон отчёта, разделяющий принятые, отклонённые и нерешённые находки, чтобы избежать ошибок с знаменателем.

Задача

A checklist built from known defects produced many false positives when applied to new guards, leading to misleading precision metrics and unclear reporting.

Результат

The audit revealed that the checklist’s precision was 44.4% on resolved findings, not 56% as previously claimed. Refining criteria reduced false positives but did not conclusively prove improvement due to sample changes. A reporting pattern that separates accepted, rejected, and unresolved findings prevents denominator mistakes.

Рабочий процесс

• 1. Run AI agents to audit guards and record findings and adjudications.
• 2. Reconstruct counts of accepted and rejected findings from audit logs.
• 3. Calculate precision as accepted/(accepted+rejected).
• 4. Identify over‑broad criteria (e.g., empty‑input) and refine contracts.
• 5. Apply tightened criteria to a new guard sample and audit again.
• 6. Use a Python script to report counts and precision, keeping unresolved findings separate.

Доказательства

A checklist built from seven known defects caught all seven, which only proves it can describe the failures that shaped it.

Источник

For the first audit, 8/18 findings were accepted: 44.4% precision under the recorded labels.

Источник

The old wording collapsed those cases. A better contract asks what the program has promised to verify.

Источник