I built Klarion, a secret scanner that asks an LLM before it alerts

Description and use

Klarion, a secret scanner that queries an LLM before alerting, cut false positives 33× compared to gitleaks and over 1,000× compared to detect‑secrets, while maintaining high precision and recall on real secrets.

Problem

Secret scanners generate many false positives, especially in typed languages, causing developers to ignore alerts and miss real leaks.

Outcome

Klarion reduced alerts by 33× compared to gitleaks and >1,000× compared to detect‑secrets, while achieving 1.7× the recall of gitleaks at similar precision.

Workflow

• Detect candidates using keyword gate, 81 regex rules, and normalized Rényi entropy.
• For each candidate, an LLM reads the surrounding code and returns "secret", "false positive", or "uncertain" with a one‑line reason.
• If the LLM flags a secret, Klarion raises an alert; otherwise it suppresses it.

Evidence

Klarion + model 11 alerts on the 4 repos, 0.18 per 1k files, 33x fewer alerts than gitleaks and over 1,000x fewer than detect‑secrets

Source

After it, Klarion's rate is 0.18 per 1k files on the typed repos and 0.20 on the control, so the language effect is gone.

Source

Klarion + model 0.35 recall, 0.89 precision; gitleaks 0.21 recall, 0.91 precision; detect‑secrets 0.35 recall, 0.34 precision.

Source