I benchmarked my own AI coding skill across 424 runs. It failed.

Описание и применение

Автор создал «Receipts» – код‑скилл, заставляющий агента предоставлять вывод команд перед заявлением о завершении. В 424 испытаниях он увеличил количество доказательств в 30‑раз, но не снизил частоту ложных успехов (≈75 %) в сложных тестах с кросс‑файловыми зависимостями.

Задача

AI coding agents often claim to have fixed bugs without actually running tests, leading to false success and unreliable code changes.

Результат

Receipts dramatically increased evidence provision but had no measurable impact on reducing false-success rates in trap-tier tests; false-success remained around 75% versus 73.6% baseline.

Рабочий процесс

• 1. Define six evidence questions (command, exit code, stdout, stderr, edited files, read files).
• 2. Implement Receipts as a Claude Code skill via SKILL.md.
• 3. Create a benchmark harness with held‑out tests, trap tier, and multiple prompt arms.
• 4. Commit predictions and run 424 trials across baseline, oneliner, persona, receipts, and variants.
• 5. Record JSON transcripts and commit them to the repo.
• 6. Analyze evidence rates and false-success percentages per arm.
• 7. Iterate on prompt designs and re-run additional trials.
• 8. Publish results and repository for replication.

Доказательства

The skill does exactly one of the two things it was supposed to do: agents paste real command output 30x more often.

Источник

At the tier where the trap fires, the skill's false-success rate is 75%. Without any prompt at all, 73.6%. That difference is inside the noise, and it does not move in the right direction.

Источник