Agent security has two failure modes: miss the attack, or break the work.

Description and use

SolonGate achieved the highest harmful‑call detection while minimizing disruption to legitimate agent work.

Problem

How to stop dangerous tool calls without breaking legitimate work in AI agents.

Outcome

SolonGate achieved the highest harmful‑call detection (73.7%) while maintaining low false‑positive rate (0.78%) and low session disruption (15%). It also had the highest MCC (0.790) and fastest latency (0.01 ms per call).

Workflow

• Collect 1,652 harmful tool calls across 15 attack families.
• Collect 24,911 benign tool calls from real agent sessions and public repositories.
• Run each guardrail (SolonGate, Claude Code permissions, Invariant, llm-guard, custom allowlists/denylists) on all calls.
• Measure harmful calls caught, legitimate calls falsely blocked, sessions broken, MCC, and latency per call.
• Publish methodology, misses, false blocks, and limitations.
• Open‑source the Agent Security Gateway.

Evidence

We ran: 1,652 harmful tool calls across 15 attack families. 24,911 benign tool calls from real agent sessions and public repositories.

Source

The result for SolonGate: 73.7% of harmful calls caught, 0.78% of legitimate calls falsely blocked, 15% of real sessions broken, MCC: 0.790, 0.01 ms per call, Nearly 3× the MCC of the next guard in the benchmark.

Source

We published the methodology, the misses, the false blocks, and the limitations too.

Source