Our cyber security scan for smart contracts found 14 of 20 known bugs. Recount it yourself, and see where it falls short

Description and use

The commercial Solidity security scanner found 14 of 20 known high‑severity bugs in five public audits, verified by a public script.

Problem

Assess the effectiveness of a commercial smart‑contract security scan against a benchmark of known high‑severity findings.

Outcome

The commercial scan achieved 14 of 20 known high findings, outperforming the older route (8) but slightly below the leader’s 15 (16 under the benchmark). The public script confirms the counts and allows independent verification.

Workflow

• Run the scan on each audit repository using the shop’s code, model and reasoning limits
• Collect the verdict files from the shop’s output
• Execute the public script node check.mjs to recompute counts and verify hashes
• Compare the counts with the EVMbench benchmark and the leader’s report
• Report discrepancies via the GitHub issue tracker

Evidence

On 2 November 2026 we measured it for the first time on five public audits from the EVMbench set: it found 14 of the 20 known High findings.

Source

The folder with the verdicts, the exact reports the judge read, a table and a script that recounts everything is public: https://github.com/holistis/bug-bounty-intelligence-mcp/tree/main/results/2026-11-02-shop-setup-five-audits

Source

The five scans cost 5.88 dollars in total.

Source