What AI code reviewers miss, and the checks I built for it

说明与用途

Healthkit 是一个 Python 代码审查工具包，测量规则效果，专注于模式检测，并通过 Schemathesis 补充行为检查，提高 AI 生成代码审查的可靠性。

问题

AI-generated code often contains subtle bugs that automated reviewers miss, leading to regressions and security issues.

结果

Healthkit reduced false positives, caught real bugs, and clarified that a clean run means "nothing it knows about" rather than safety.

工作流

• 1. Measure rule effectiveness on a human-reviewed module (45 findings, 9 true positives).
• 2. Design rules around patterns, project knowledge, and behavior.
• 3. Build healthkit with deterministic rules, AI advisory only.
• 4. Run healthkit review on changed lines with "healthkit review --base main".
• 5. Flag any rule or # noqa changes for human approval.
• 6. Use healthkit api to test API routes for crashes and access leaks.
• 7. Iterate rules based on measured precision and coverage.

证据

A careful human review of one module had produced 45 findings.

来源

The first score was 29 of 45. It looked great. Then we read every hit by hand. Most were coincidences: an unused import that happened to sit inside a behaviour fix, findings inside routes the fix deleted anyway. Counting only the hits where the rule pointed at the actual problem: 9 of 45, about 20%.

来源

healthkit can never judge whether the logic is right, whether it is secure, or whether the design is right. A clean run means "nothing it knows about", never "safe".

来源

healthkit api sends good and bad input to every route of an API (via Schemathesis) and treats any server error as a finding. On the test project it found six real crashes, none of which had a rule.

来源