Stop Writing Agent Prompts for Deterministic Work

Description and use

The npm CLI version check was rewritten from an agent loop to a script to eliminate silent failures and improve monitoring reliability.

Problem

A scheduled job that checks for newer npm CLI versions failed 29 times in a row because it silently exited when the global model changed, making failures indistinguishable from healthy runs.

Outcome

The deterministic version check now runs reliably, reporting failures immediately and never silently succeeding. The overall reliability of the monitoring fleet improved, and silent failures are now detectable via execution logs.

Workflow

• 1. Identify deterministic checks that only compare values.
• 2. Rewrite the check as a script with explicit success/failure exits.
• 3. Ensure exit 0 prints a success message and exit 1 prints an error and never remains silent.
• 4. Replace agent loops with scripts for these checks.
• 5. Keep only non‑deterministic loops (email importance, voucher overspend, project stalled) as agent loops, adding a verifier if needed.
• 6. Monitor runner execution logs for silent failures.
• 7. Adjust notification logic to treat exit 1 as loud and exit 0 as silent unless configured otherwise.

Evidence

One of my scheduled jobs failed 29 times in a row. Nobody told me, because it was designed to speak only when it had something to report — and a run that dies before it can speak looks exactly like a run that found nothing wrong.

Source

The fix was not a better prompt. It was deleting the model from the loop.

Source

If both answers are a number or a string, it is a script. If the answer is "it depends what it says", it needs a model — and then it needs a verifier I trust more than the agent, because I am no longer able to predict the output.

Source