Detecting Silent Failures in Unattended AI Trading Bots

Описание и применение

Разработчик обнаружил, что его торговый бот на Python крашился в течение трех недель из-за ошибки аутентификации, но планировщик задач Windows сообщал об успехе (exit code 0). Стандартные инструменты мониторинга не выявили проблему, так как обертка процесса работала корректно. Это подчеркивает необходимость мониторинга бизнес-результатов, а не только жизнеспособности процесса.

Задача

Standard monitoring tools (exit codes, dead-man switches) failed to detect that an unattended AI trading bot was crashing internally due to authentication errors for three weeks, while the wrapper process reported success.

Результат

The developer discovered that 18,000+ tracebacks were ignored because the wrapper process reported success. This highlights the insufficiency of exit-code-based monitoring for complex AI agents and the need for outcome-based validation.

Рабочий процесс

• Run unattended scheduled tasks (e.g., trading bots) via OS scheduler (Windows Task Scheduler).
• Implement a wrapper process that handles execution and reports exit codes.
• Use standard uptime monitoring (dead-man's switch) to verify the wrapper is running.
• Identify failure mode: Wrapper exits 0, but internal logic fails (e.g., auth error), resulting in no actual work done.
• Manually inspect raw log files to discover accumulated tracebacks and lack of expected outputs (e.g., trade records).
• Design new monitoring logic that validates specific business outcomes (e.g., 'did a trade occur?') rather than just process health.

Доказательства

A different bot's scheduled task kept reporting success — exit code 0, every 5-minute run, for over three weeks straight... Inside, the actual Python process had been crashing on nearly every cycle that whole time

Источник

The classic dead-man's-switch tools... would have shown green the entire three weeks, too — the wrapper process itself never stopped running or stopped pinging.

Источник