My agent looked fine — until I gave it an empty list

Описание и применение

Пользователь создал инструмент для проверки поведения AI-агента за 10 минут вместо доверия тому, что агент «имеет право». Инструмент оценивает 4 фактора на задачу: завершена, верна, не fabricровала, не совершила ошибку. Фиксированный ответ на пустой список набрал 4/4, оригинальная функция — 3/4.

Задача

An AI agent can appear correct on a task but still fail when edge cases are introduced, so a quick automated check is needed to catch silent failures before trusting the agent's output.

Результат

The fixed answer scored 4/4 (finished, correct, did not make things up, took no wrong action); the original unfixed function scored 3/4 and lost the 'Is correct' point.

Рабочий процесс

• Paste your agent's answer into the tool at https://jefferson999.github.io/agent-reliability--score/
• Run the score and read what to fix
• If the agent passes the empty-list task, try the other 19 tasks
• Use the tool as a quick first check, not as a CI or production replacement

Доказательства

One test from it: 'Handle an empty average.' The starting code divides by the list length, so an empty list breaks it.

A fixed answer — check for an empty list first and return 0, otherwise return the normal average — scored 4/4. What-to-fix: 'No fix needed for this task.'

The original, unfixed function scored 3/4. It lost the 'Is correct' point. The plain-language fix: tell the agent to check for an empty list first and return 0, and keep the normal average for everything else.

It scores 4 things per task: finished, correct, did not make things up, and took no wrong action. Auto-score suggests, you confirm — so you get speed without handing the judgment to a black box.