We ran 150 real AI agent outputs through COGEXT. Here’s what we found.

Описание и применение

Автоматический аудит 150 реальных выводов AI‑агентов показал, что лишь 0,8 % содержат подтверждаемые обещания, а большинство обещаний в специально отобранных примерах не указывают получателя и срок, что делает их непроверяемыми.

Задача

AI agents often make promises but there is no system to track whether those promises are kept, and many promises lack verifiable details such as recipients and deadlines.

Результат

Out of 120 unbiased samples, only 1 commitment was found (0.8%). In a promise‑enriched set of 30 samples, 13 commitments were extracted; 61.5% lacked recipients and 92.3% lacked deadlines, rendering them unverifiable despite high confidence. The audit demonstrates that many agent promises are incomplete and cannot be automatically tracked.

Рабочий процесс

• Collect 150 agent outputs from public sources (GitHub notebooks, DEV.to articles, Hacker News comments).
• POST each sample to COGEXT /api/v1/ingest with 1‑second delay.
• Analyze extracted commitments: count, recipient presence, deadline presence, classification, confidence, verifiability status.
• Generate verifier queries for each commitment.
• Compile findings and publish audit results.
• Provide dataset and scripts for reproducibility.
• Invite users to run audit on their own agents.

Доказательства

Out of 120 randomly collected agent outputs, COGEXT extracted 1 commitment.

Источник

61.5% had no recipient named.

Источник

92.3% had no deadline.

Источник