I benchmarked my own AI coding skill across 424 runs. It failed.

说明与用途

作者创建了Receipts技能，强制AI代理在声明完成前提供命令输出。在424次运行中，该技能将证据提供率提高了30倍，但在陷阱级测试中未能降低错误成功率（约75%）。

问题

AI coding agents often claim to have fixed bugs without actually running tests, leading to false success and unreliable code changes.

结果

Receipts dramatically increased evidence provision but had no measurable impact on reducing false-success rates in trap-tier tests; false-success remained around 75% versus 73.6% baseline.

工作流

• 1. Define six evidence questions (command, exit code, stdout, stderr, edited files, read files).
• 2. Implement Receipts as a Claude Code skill via SKILL.md.
• 3. Create a benchmark harness with held‑out tests, trap tier, and multiple prompt arms.
• 4. Commit predictions and run 424 trials across baseline, oneliner, persona, receipts, and variants.
• 5. Record JSON transcripts and commit them to the repo.
• 6. Analyze evidence rates and false-success percentages per arm.
• 7. Iterate on prompt designs and re-run additional trials.
• 8. Publish results and repository for replication.

证据

The skill does exactly one of the two things it was supposed to do: agents paste real command output 30x more often.

来源

At the tier where the trap fires, the skill's false-success rate is 75%. Without any prompt at all, 73.6%. That difference is inside the noise, and it does not move in the right direction.

来源