My Linter Scored 26% on Its Home Corpus and 0% on Docs — the Calibration Rule I Ship With Now

Описание и применение

Валидация линтера, успешно проходившая тесты, показывала низкую точность на реальном коде и документации. Внедрена система из трёх проверок: отсутствие HIGH‑сообщений, объединение MEDIUM‑сообщений в один класс и отдельный код выхода при отсутствии входных данных.

Задача

A linter validator that performed well on its own test fixtures but produced high false positives on real code and documentation, revealing a calibration issue.

Результат

The new calibration gate reduced false positives, ensuring validators only publish when they have zero HIGH findings on real code, a single MEDIUM class, and proper handling of no input.

Рабочий процесс

• Run validator on real, non‑author corpus
• Check that HIGH=0; if not, reject package
• Collapse MEDIUM findings into one rule class; if multiple, reject
• Ensure NO_INPUT returns exit code 3 and NO_INPUT verdict
• Add counters for unknown licenses and other false positives
• Publish only if all gates pass

Доказательства

My cron validator parses schedules out of five different formats: crontab, GitHub Actions schedule, systemd .service/.timer, and a JSON job file.

Источник

The rule did not have a dialect problem. It had a lexing problem, and I had been about to fix it in the wrong place.

Источник

1. Real corpus, HIGH = 0.

Источник

1. Zero input gets its own exit code.

Источник