A token‑risk check for AI agents that publishes its own error rates

Description and use

VetAgent demonstrates a transparent, reproducible token‑risk assessment with publicly available error rates, reducing false positives and exposing weaknesses that can be addressed through continuous benchmarking.

Problem

AI agents need a reliable way to assess token risk before buying, but existing scanners lack transparent, self‑published error rates and often misclassify tokens, leading to false positives and missed threats.

Outcome

VetAgent demonstrates a transparent, reproducible token‑risk assessment with publicly available error rates, reducing false positives and exposing weaknesses that can be addressed through continuous benchmarking.

Workflow

• 1. Clone the VetAgent repo and run the benchmark harness (bench/run_benchmark.py).
• 2. The harness fetches token data from upstream sources and compares VetAgent’s assessments to the frozen labels in bench/dataset.json.
• 3. If any verdict diverges from the label, the harness exits non‑zero, ensuring continuous validation.
• 4. Deploy the MCP server (https://vetagent.dev/mcp) or call the HTTP endpoint GET /assess/<address> to assess new tokens.
• 5. Monitor the published error rates and update the dataset as new adversarial or benign tokens are identified.