I Audited Every Decision My AI Agents Made for 30 Days

Описание и применение

Проверка решений автономных ИИ‑агентов выявила скрытые ошибки. Внедрение журнала решений, отдельного отчёта, очереди проверки и ручного обзора снизило критические ошибки до минимуму.

Задача

Unattended AI agents made incorrect or hallucinated decisions that were not caught by existing monitoring, leading to customer complaints and hidden failures.

Результат

After 30 days, critical errors dropped from 38 to single digits; monitoring became honest and actionable.

Рабочий процесс

• 1. Add decision log to each agent action. 2. Run agents overnight. 3. Generate morning digest from raw log (no LLM). 4. Queue decisions with blast_radius external/money and confidence <0.9. 5. Manually review 10 random decisions daily. 6. Flag silent retries after 3 attempts. 7. Adjust thresholds and rules as needed.

Доказательства

Most agent setups log events — "sent email", "ran script", "finished job". That's a diary of outcomes, not decisions.

Источник

Four fields did most of the work: input_digest, confidence, blast_radius, result.

Источник

The honest failure: I was grading homework written by the student.

Источник

Four changes, all low-tech: separate reporter, review queue, random sample, silent retries loud.

Источник