Designing a compliance check that does not cry wolf in CI

Description and use

Dependency scanners often silently drop unparseable versions, producing incomplete reports. Replacing re.match with re.search, adding comprehensive tests, and using distinct exit codes and file‑change filters for pre‑commit hooks resolve the issue and improve CI reliability.

Problem

Compliance scanners often silently drop unparseable version strings, producing plausible but incomplete reports that lead to disabled checks and ignored security issues.

Outcome

The updated scanner reliably reports all supported dependency versions, reduces false negatives, and prevents unnecessary disabling of CI checks.

Workflow

• 1. Identify the parsing bug by testing against real ecosystem inputs.
• 2. Replace re.match with re.search to find the first numeric component.
• 3. Add comprehensive parameterized tests covering all real-world version strings.
• 4. Log the number of dependencies processed and the number successfully parsed to surface silent drops.
• 5. Implement three exit codes: 0 (clean), 1 (issues found), 2 (no manifests).
• 6. Configure CI to treat exit code 2 as non‑fatal or as a policy decision.
• 7. Use pre‑commit hooks with file‑change filters to avoid blocking commits in repos without relevant files.

Evidence

The first version of that helper used this: re.match(r'^(d+)(?:.(d+))?')… The match fails… The component is dropped – silently, with no warning, no error and no log line.

Source

The fix is one word: m = re.search(r'(d+)(?:.(d+))?')… Note lts/gallium still yields nothing, correctly.

Source