My Code Reviewer Scored a Nonexistent Directory 100/100 and Exited 0

Описание и применение

Проверка инструментов CI выявила, что некоторые скрипты возвращают успешный статус при пустом или несуществующем пути, что приводит к ложным подтверждениям. Автор предлагает стандартизировать команды self-test и проверять реальную работу, а не только код возврата.

Задача

A CI gate that relies on a tool’s exit status can falsely approve code when the tool reports success on an empty or nonexistent input path.

Результат

The audit revealed that tools can falsely pass when given empty or nonexistent input, leading to unsafe CI gates. The author recommends standardizing self-test contracts, asserting on actual work, and hard-failing on empty input.

Рабочий процесс

• Run audit script over all skill packages
• Detect packages with selftest subcommand or --selftest flag
• Invoke each package’s self-test correctly
• Check exit code and output
• Identify false negatives from incorrect invocation
• Find tool that reports 100/100 on nonexistent path
• Explain CI implications
• Propose five rules for reliable gates

Доказательства

One of those five is a code-review tool. Called with a positional argument, it does not complain. It treats the argument as a path:

Источник

skipped nonexistent path: selftest # Code review report - scope: 0 files, 0 lines - findings: 0 (critical 0 / high 0 / medium 0 / low 0) - health score: 100/100 — ✅ safe to merge EXIT=0

Источник

12 packages passed
5 packages "failed"
17 packages had no self-test at all

Источник