My Code Reviewer Scored a Nonexistent Directory 100/100 and Exited 0

说明与用途

审计发现 CI 工具在给定空或不存在路径时可能错误地报告成功，导致不安全的合并。作者建议标准化 self-test 命令、断言实际工作并在空输入时硬失败。

问题

A CI gate that relies on a tool’s exit status can falsely approve code when the tool reports success on an empty or nonexistent input path.

结果

The audit revealed that tools can falsely pass when given empty or nonexistent input, leading to unsafe CI gates. The author recommends standardizing self-test contracts, asserting on actual work, and hard-failing on empty input.

工作流

• Run audit script over all skill packages
• Detect packages with selftest subcommand or --selftest flag
• Invoke each package’s self-test correctly
• Check exit code and output
• Identify false negatives from incorrect invocation
• Find tool that reports 100/100 on nonexistent path
• Explain CI implications
• Propose five rules for reliable gates

证据

One of those five is a code-review tool. Called with a positional argument, it does not complain. It treats the argument as a path:

来源

skipped nonexistent path: selftest # Code review report - scope: 0 files, 0 lines - findings: 0 (critical 0 / high 0 / medium 0 / low 0) - health score: 100/100 — ✅ safe to merge EXIT=0

来源

12 packages passed
5 packages "failed"
17 packages had no self-test at all

来源