I Stopped Trusting AI Agents With Tools. So I Built a Gatekeeper

Description and use

The author built Agent ToolTrust, a contextual risk engine for AI agent tools, field‑tested it on 83 real agents across 10 frameworks without mocks, achieving 100% pass rate.

Problem

AI agents often have binary allow/deny permissions, leading to over-privileged actions or approval fatigue, and mock agents hide real integration issues.

Outcome

Agent ToolTrust enabled safe, auditable tool usage across 10 frameworks with 83 real agents, achieving 100% pass rate in field tests and uncovering 7 critical failures that would have been missed with mock agents.

Workflow

• Define policy and architecture before coding
• Build adapters for 10 agent frameworks
• Source 83 real agents from GitHub
• Create 30 scenarios covering all decision types and adversarial cases
• Run field tests with real agents and a local LLM
• Collect and analyze decisions and failures
• Iterate on adapters until all 83 agents pass all scenarios
• Publish engine and adapters to PyPI
• Document design decisions and field test results

Evidence

83/83 agents passed. 2,490 tests green.

Source

Zero mock agents. 83 real ones across 10 frameworks.

Source