We benchmarked a security detector against 500k lines of Go. It found a real bug.

Description and use

A security detector was benchmarked on 500k lines of Go, finding a real token comparison bug and proving its precision for CI gating.

Problem

Ensuring that security detectors for Go code correctly identify credential comparison bugs without generating false positives across large codebases.

Outcome

A precision‑focused detector identified a single real timing side‑channel bug in a large Go codebase, proving that source‑tracking outperforms name‑matching and that zero findings on 430k LOC is a strong indicator of detector quality.

Workflow

• 1. Define rule: secrets must be compared with crypto/subtle.ConstantTimeCompare. 2. Implement detector to fire on == comparisons where an operand originates from a credential source (e.g., os.Getenv, r.Header.Get). 3. Exclude test files, vendor code, and resolve helper functions one level deep. 4. Benchmark detector against 8 open‑source Go repos (~503k LOC). 5. Analyze results: 0 findings in 7 repos, 1 real finding in goreleaser. 6. Review finding, confirm it is a timing side‑channel risk. 7. Submit PR to goreleaser to replace == with ConstantTimeCompare. 8. Verify PR acceptance and record outcome.

Evidence

The detector only fires when an operand has a traceable credential source — a value that came from r.Header.Get(...), os.Getenv(...), or an assignment chained from one of those.

Source

It produced exactly one finding. The finding was real.

Source

The PR is here: https://github.com/goreleaser/goreleaser/pull/6813.

Source