I Stopped Trusting AI Agents With Tools. So I Built a Gatekeeper

说明与用途

作者构建了 Agent ToolTrust，一种 AI 代理工具的上下文风险引擎，在 10 个框架的 83 个真实代理上进行现场测试，无需使用模拟器，成功率达到 100%。

问题

AI agents often have binary allow/deny permissions, leading to over-privileged actions or approval fatigue, and mock agents hide real integration issues.

结果

Agent ToolTrust enabled safe, auditable tool usage across 10 frameworks with 83 real agents, achieving 100% pass rate in field tests and uncovering 7 critical failures that would have been missed with mock agents.

工作流

• Define policy and architecture before coding
• Build adapters for 10 agent frameworks
• Source 83 real agents from GitHub
• Create 30 scenarios covering all decision types and adversarial cases
• Run field tests with real agents and a local LLM
• Collect and analyze decisions and failures
• Iterate on adapters until all 83 agents pass all scenarios
• Publish engine and adapters to PyPI
• Document design decisions and field test results

证据

83/83 agents passed. 2,490 tests green.

来源

Zero mock agents. 83 real ones across 10 frameworks.

来源