I Audited Every Decision My AI Agents Made for 30 Days

Description and use

Auditing autonomous AI agents uncovered hidden errors. Implementing a decision log, separate digest, review queue, and manual checks reduced critical mistakes to a minimum.

Problem

Unattended AI agents made incorrect or hallucinated decisions that were not caught by existing monitoring, leading to customer complaints and hidden failures.

Outcome

After 30 days, critical errors dropped from 38 to single digits; monitoring became honest and actionable.

Workflow

• 1. Add decision log to each agent action. 2. Run agents overnight. 3. Generate morning digest from raw log (no LLM). 4. Queue decisions with blast_radius external/money and confidence <0.9. 5. Manually review 10 random decisions daily. 6. Flag silent retries after 3 attempts. 7. Adjust thresholds and rules as needed.

Evidence

Most agent setups log events — "sent email", "ran script", "finished job". That's a diary of outcomes, not decisions.

Source

Four fields did most of the work: input_digest, confidence, blast_radius, result.

Source

The honest failure: I was grading homework written by the student.

Source

Four changes, all low-tech: separate reporter, review queue, random sample, silent retries loud.

Source