Agent security has two failure modes: miss the attack, or break the work.

说明与用途

SolonGate 在检测有害工具调用方面表现最佳，同时对合法代理工作干扰最小。

问题

How to stop dangerous tool calls without breaking legitimate work in AI agents.

结果

SolonGate achieved the highest harmful‑call detection (73.7%) while maintaining low false‑positive rate (0.78%) and low session disruption (15%). It also had the highest MCC (0.790) and fastest latency (0.01 ms per call).

工作流

• Collect 1,652 harmful tool calls across 15 attack families.
• Collect 24,911 benign tool calls from real agent sessions and public repositories.
• Run each guardrail (SolonGate, Claude Code permissions, Invariant, llm-guard, custom allowlists/denylists) on all calls.
• Measure harmful calls caught, legitimate calls falsely blocked, sessions broken, MCC, and latency per call.
• Publish methodology, misses, false blocks, and limitations.
• Open‑source the Agent Security Gateway.

证据

We ran: 1,652 harmful tool calls across 15 attack families. 24,911 benign tool calls from real agent sessions and public repositories.

来源

The result for SolonGate: 73.7% of harmful calls caught, 0.78% of legitimate calls falsely blocked, 15% of real sessions broken, MCC: 0.790, 0.01 ms per call, Nearly 3× the MCC of the next guard in the benchmark.

来源

We published the methodology, the misses, the false blocks, and the limitations too.

来源