My Code Reviewer Scored a Nonexistent Directory 100/100 and Exited 0

Description and use

The audit uncovered that CI tools can falsely report success when given empty or nonexistent input, leading to unsafe merges. The author recommends standardizing self-test contracts, asserting on actual work, and hard-failing on empty input.

Problem

A CI gate that relies on a tool’s exit status can falsely approve code when the tool reports success on an empty or nonexistent input path.

Outcome

The audit revealed that tools can falsely pass when given empty or nonexistent input, leading to unsafe CI gates. The author recommends standardizing self-test contracts, asserting on actual work, and hard-failing on empty input.

Workflow

• Run audit script over all skill packages
• Detect packages with selftest subcommand or --selftest flag
• Invoke each package’s self-test correctly
• Check exit code and output
• Identify false negatives from incorrect invocation
• Find tool that reports 100/100 on nonexistent path
• Explain CI implications
• Propose five rules for reliable gates

Evidence

One of those five is a code-review tool. Called with a positional argument, it does not complain. It treats the argument as a path:

Source

skipped nonexistent path: selftest # Code review report - scope: 0 files, 0 lines - findings: 0 (critical 0 / high 0 / medium 0 / low 0) - health score: 100/100 — ✅ safe to merge EXIT=0

Source

12 packages passed
5 packages "failed"
17 packages had no self-test at all

Source