I Stopped Trusting AI Agents With Tools. So I Built a Gatekeeper

Description and use

A deterministic permission engine for AI agents was built, evaluating tool calls with contextual risk assessment. Field testing with 83 real agents across 10 frameworks validated its effectiveness, preventing unauthorized actions.

Problem

AI agents often have binary allow/deny permissions, leading to over-privileged actions or approval fatigue, and mock agents hide real integration problems.

Outcome

Implemented a deterministic, contextual permission engine that prevented unauthorized tool usage across 83 real agents and 10 frameworks, achieving 100% pass rate in field tests and exposing 7 real integration bugs that would have been missed with mock agents.

Workflow

• Define policy and architecture before coding
• Build adapters for 10 agent frameworks
• Create 30 decision and adversarial scenarios
• Run 83 real agents through 2,490 test cases
• Collect audit logs and decision explanations
• Iterate on adapters until all pass
• Publish engine and adapters to PyPI
• Deploy with shadow mode for tuning
• Enforce policy in production
• Monitor and audit decisions

Evidence

83 real agents across 10 frameworks. A covering design that cut a 12-day test matrix into one afternoon.

Source

2,490 tests green. 83/83 agents passed.

Source

The engine is deterministic. The LLM proposes, policy disposes.

Source