I scored the tool descriptions of Chrome DevTools MCP and Playwright MCP. Both rarely say when to use a tool

Описание и применение

Описание инструментов в серверах браузерной автоматизации часто не указывает, когда использовать инструмент и что он возвращает, что затрудняет выбор AI‑агента. Скрипт Forecall оценивает описания, выявляя пробелы. Переписывание низкооцененных описаний повышает баллы, но может ухудшить точность некоторых моделей.

Задача

Tool descriptions in browser‑automation MCP servers often omit when to use a tool and what it returns, making it hard for AI agents to choose correctly.

Результат

Tool descriptions in both MCP servers score poorly on usage guidance and return value. Adding concise, specific sentences can dramatically improve scores, but may negatively impact model accuracy for some agents.

Рабочий процесс

• 1. Run each MCP server locally and capture tools/list JSON.
• 2. Use Forecall CLI (npx forecall lint) to score each tool on six parts: Purpose, When to use, Arguments, Return value, Constraints and side effects, Examples.
• 3. Analyze score distribution; identify gaps (missing when‑to‑use, return value).
• 4. Rewrite selected low‑scoring tool descriptions adding purpose, usage, constraints, return value and example.
• 5. Re‑score rewritten descriptions to confirm score improvement.
• 6. Test real models (Claude, GPT, Codex) on one‑step tasks to evaluate tool‑selection accuracy with original vs rewritten descriptions.
• 7. Record model performance and note any regressions.
• 8. Publish results and provide instructions for users to run Forecall on their own servers.
• 9. Encourage iterative rewriting and real‑model testing to close gaps.

Доказательства

Chrome DevTools MCP, 24 of 30 tools never say when to use them; in Playwright MCP, 23 of 25.

Источник

I rewrote drag's description using only facts from its schema. I added three things: when to use it (and how it differs from upload_file), what to do before calling it, and what it returns.

Источник

With the descriptions as they are, no model picked a wrong tool: Claude Code with Fable 5.1, Sonnet 5.5 and Haiku 4.5 (24 of 24)... The only wrong picks, click instead of hover from two smaller Codex models, came from a task whose text was ambiguous.

Источник