My Linter Scored 26% on Its Home Corpus and 0% on Docs — the Calibration Rule I Ship With Now

Description and use

A linter that passed self‑tests but had many false positives on real code and docs. Implemented a three‑step gate: zero HIGH findings on real code, single MEDIUM class, and NO_INPUT exit code.

Problem

A linter validator that performed well on its own test fixtures but produced high false positives on real code and documentation, revealing a calibration issue.

Outcome

The new calibration gate reduced false positives, ensuring validators only publish when they have zero HIGH findings on real code, a single MEDIUM class, and proper handling of no input.

Workflow

• Run validator on real, non‑author corpus
• Check that HIGH=0; if not, reject package
• Collapse MEDIUM findings into one rule class; if multiple, reject
• Ensure NO_INPUT returns exit code 3 and NO_INPUT verdict
• Add counters for unknown licenses and other false positives
• Publish only if all gates pass

Evidence

My cron validator parses schedules out of five different formats: crontab, GitHub Actions schedule, systemd .service/.timer, and a JSON job file.

Source

The rule did not have a dialect problem. It had a lexing problem, and I had been about to fix it in the wrong place.

Source

1. Real corpus, HIGH = 0.

Source

1. Zero input gets its own exit code.

Source