I benchmarked my own security tool against 3 others — and wrote down what it lost

Description and use

gedik matched or exceeded other tools in detection on the custom target, but suffered delivery failures on the NodeGoat target, higher cost, and lacked machine‑readable output. It uniquely performed mutation testing of project tests, a feature absent in competitors.

Problem

Evaluating the performance and limitations of a new LLM‑based security audit tool (gedik) compared to existing tools, focusing on detection, delivery, cost, and output quality.

Outcome

gedik matched or exceeded other tools in detection on the custom target, but suffered delivery failures on the NodeGoat target, higher cost, and lacked machine‑readable output. It uniquely performed mutation testing of project tests, a feature absent in competitors.

Workflow

• 1. Prepare four tools: gedik v2.6.0, cloudflare/security‑audit‑skill, Anthropic’s /security-review, Semgrep 1.178.0.
• 2. Define two targets: NodeGoat (c5cb68a) with 17 known issues, and a custom Supabase+LLM repo with 11 seeded issues and 5 decoys.
• 3. Run each tool on each target on 30 Sep 2026, recording detection, false positives, report delivery, and cost.
• 4. Repeat gedik runs on 1 Oct 2026 after account approval to assess impact of account level changes.
• 5. Collect and tabulate results, noting delivery failures, cost differences, and unique mutation‑testing outcomes.
• 6. Analyze losses: error messages, missing JSON output, slower execution, and delivery dependence on account approval.
• 7. Summarize findings and publish README with balanced claims.
• 8. Invite a second reader to validate the benchmark.
• 9. Publish source code and run records on GitHub.
• 10. Encourage future users to measure delivery, not just detection.