I invented a CVE number to test my tool. It was real

Description and use

A tool automatically verifies claimed identifiers in security reports against a repository, but mistakenly confirms a real CVE, underscoring the need for contextual validation.

Problem

Maintainers receive bug reports with fabricated identifiers that exist in other projects, making triage costly.

Outcome

The tool identified fabricated claims accurately but misidentified a real CVE, highlighting the need for contextual checks. Benchmarking on curl advisories reduced unexplained misses from 24% to 3.1% after parser fixes.

Workflow

• 1. Extract checkable claims from a security report.
• 2. Resolve each claim against the repository at the specified revision.
• 3. Report claims that do not resolve.
• 4. Run the tool against a fabricated report to validate end‑to‑end.
• 5. Measure performance on real advisories (curl OSV) and documentation corpus.

Evidence

The tool came back and told me the CVE was real. It's a Linux kernel TPM bug — "st33zp24: Fix missing cleanup on get_burstcount() error."

Source

Fixing both took the pinned advisory row from 24.0% unexplained to 3.1%.

Source