My AI Agent Kept Calling the Wrong Tool

Описание и применение

Переписав шесть строк JSON, описывающих инструменты, автор снизил частоту ошибок агента с ~20% до менее 2%, не меняя модель и не увеличивая расходы.

Задача

The support agent repeatedly called the wrong tool, returning irrelevant policy answers instead of order status, causing a ~20% failure rate.

Результат

Rewriting six lines of JSON tool definitions reduced the agent’s failure rate from ~20% to <2%, eliminating the need for model upgrades or cost increases.

Рабочий процесс

• 1. Identify wrong tool usage via conversation logs.
• 2. Log each tool call with timestamp, name, arguments, and preceding message.
• 3. Analyze logs to find patterns of mis‑calls.
• 4. Rewrite tool descriptions to specify when not to use them.
• 5. Declare prerequisites in tool descriptions (e.g., require order number).
• 6. Rename ambiguous tools to be self‑documenting.
• 7. Constrain parameters with enums or explicit formats.
• 8. Deploy updated tool schemas and test.
• 9. Monitor tool call logs to confirm improvement.

Доказательства

The failure rate went from ~20% of conversations to under 2% on the day I shipped the rewrite.

Источник

The bug was in 6 lines of JSON I wrote in a hurry and never revisited.

Источник