Detecting Silent Failures in Unattended AI Trading Bots

Description and use

A developer found that their Python trading bot crashed for three weeks due to an auth error, but Windows Task Scheduler reported success (exit code 0). Standard monitoring tools missed the issue because the wrapper process ran correctly. This highlights the need for outcome-based monitoring rather than just process health checks.

Problem

Standard monitoring tools (exit codes, dead-man switches) failed to detect that an unattended AI trading bot was crashing internally due to authentication errors for three weeks, while the wrapper process reported success.

Outcome

The developer discovered that 18,000+ tracebacks were ignored because the wrapper process reported success. This highlights the insufficiency of exit-code-based monitoring for complex AI agents and the need for outcome-based validation.

Workflow

• Run unattended scheduled tasks (e.g., trading bots) via OS scheduler (Windows Task Scheduler).
• Implement a wrapper process that handles execution and reports exit codes.
• Use standard uptime monitoring (dead-man's switch) to verify the wrapper is running.
• Identify failure mode: Wrapper exits 0, but internal logic fails (e.g., auth error), resulting in no actual work done.
• Manually inspect raw log files to discover accumulated tracebacks and lack of expected outputs (e.g., trade records).
• Design new monitoring logic that validates specific business outcomes (e.g., 'did a trade occur?') rather than just process health.

Evidence

A different bot's scheduled task kept reporting success — exit code 0, every 5-minute run, for over three weeks straight... Inside, the actual Python process had been crashing on nearly every cycle that whole time

Source

The classic dead-man's-switch tools... would have shown green the entire three weeks, too — the wrapper process itself never stopped running or stopped pinging.

Source