I Stopped Trusting AI Agents With Tools. So I Built a Gatekeeper

Описание и применение

Разработчик создал Agent ToolTrust – систему контекстного контроля доступа для инструментов ИИ, протестировав её на 83 реальных агентах из 10 фреймворков, без использования моков, и добился 100% успешных тестов.

Задача

AI agents often have binary allow/deny permissions, leading to over-privileged actions or approval fatigue, and mock agents hide real integration issues.

Результат

Agent ToolTrust enabled safe, auditable tool usage across 10 frameworks with 83 real agents, achieving 100% pass rate in field tests and uncovering 7 critical failures that would have been missed with mock agents.

Рабочий процесс

• Define policy and architecture before coding
• Build adapters for 10 agent frameworks
• Source 83 real agents from GitHub
• Create 30 scenarios covering all decision types and adversarial cases
• Run field tests with real agents and a local LLM
• Collect and analyze decisions and failures
• Iterate on adapters until all 83 agents pass all scenarios
• Publish engine and adapters to PyPI
• Document design decisions and field test results

Доказательства

83/83 agents passed. 2,490 tests green.

Источник

Zero mock agents. 83 real ones across 10 frameworks.

Источник