I Audited Every Decision My AI Agents Made for 30 Days

说明与用途

对自主 AI 代理的决策进行审计，发现隐藏错误。通过实施决策日志、独立摘要、审核队列和人工检查，将关键错误降至最低。

问题

Unattended AI agents made incorrect or hallucinated decisions that were not caught by existing monitoring, leading to customer complaints and hidden failures.

结果

After 30 days, critical errors dropped from 38 to single digits; monitoring became honest and actionable.

工作流

• 1. Add decision log to each agent action. 2. Run agents overnight. 3. Generate morning digest from raw log (no LLM). 4. Queue decisions with blast_radius external/money and confidence <0.9. 5. Manually review 10 random decisions daily. 6. Flag silent retries after 3 attempts. 7. Adjust thresholds and rules as needed.

证据

Most agent setups log events — "sent email", "ran script", "finished job". That's a diary of outcomes, not decisions.

来源

Four fields did most of the work: input_digest, confidence, blast_radius, result.

来源

The honest failure: I was grading homework written by the student.

来源

Four changes, all low-tech: separate reporter, review queue, random sample, silent retries loud.

来源