We ran 150 real AI agent outputs through COGEXT. Here’s what we found.

Description and use

An audit of 150 real AI agent outputs found that only 0.8 % contain verifiable commitments; most promises lack recipients and deadlines, rendering them unverifiable despite high confidence.

Problem

AI agents often make promises but there is no system to track whether those promises are kept, and many promises lack verifiable details such as recipients and deadlines.

Outcome

Out of 120 unbiased samples, only 1 commitment was found (0.8%). In a promise‑enriched set of 30 samples, 13 commitments were extracted; 61.5% lacked recipients and 92.3% lacked deadlines, rendering them unverifiable despite high confidence. The audit demonstrates that many agent promises are incomplete and cannot be automatically tracked.

Workflow

• Collect 150 agent outputs from public sources (GitHub notebooks, DEV.to articles, Hacker News comments).
• POST each sample to COGEXT /api/v1/ingest with 1‑second delay.
• Analyze extracted commitments: count, recipient presence, deadline presence, classification, confidence, verifiability status.
• Generate verifier queries for each commitment.
• Compile findings and publish audit results.
• Provide dataset and scripts for reproducibility.
• Invite users to run audit on their own agents.

Evidence

Out of 120 randomly collected agent outputs, COGEXT extracted 1 commitment.

Source

61.5% had no recipient named.

Source

92.3% had no deadline.

Source