Capbroker: I gave an AI agent a fake GitHub key, then watched it get tricked into trying to delete a repo anyway

Description and use

<p>Most AI-agent setups today hand the agent the real API key directly — in an<br> environment variable, a config file, sometimes typed straight into the<br> conversation. If that agent is tricked — via prompt injection, a poisoned<br> tool description, a malicious document it reads — into misusing that key,<br> nothing stops it, because there is no separate authority checking its work.</p> <p>I wanted to see how bad this actually is, with a real model, a real attack,<br> and no simulation. So I built <strong>capbroker</strong>: a local, self-hosted broker that<br> gives an AI agent a scoped,

Problem

<p>Most AI-agent setups today hand the agent the real API key directly — in an<br> environment variable, a config file, sometimes typed straight into the<br> conversation. If that agent is tricked — via prompt injection, a poisoned<br> tool description, a malicious document it reads — into misusing that key,<br> nothing stops it, because there is no separate authority checking its work.</p> <p>I wanted to see how bad this actually is, with a real model, a real attack,<br> and no simulation. So I built <strong>capbroker</strong>: a local, self-hosted broker that<br> gives an AI agent a scoped,

Outcome

<p>Most AI-agent setups today hand the agent the real API key directly — in an<br> environment variable, a config file, sometimes typed straight into the<br> conversation. If that agent is tricked — via prompt injection, a poisoned<br> tool description, a malicious document it reads — into misusing that key,<br> nothing stops it, because there is no separate authority checking its work.</p> <p>I wanted to see how bad this actually is, with a real model, a real attack,<br> and no simulation. So I built <strong>capbroker</strong>: a local, self-hosted broker that<br> gives an AI agent a scoped,

Evidence

Capbroker: I gave an AI agent a fake GitHub key, then watched it get tricked into trying to delete a repo anyway

Source