I built three tools to audit MCP servers. Each one found a bug in itself first.

Описание и применение

Созданы три CLI-инструмента для аудита серверов MCP: mcp-doctor (статический анализ), mcp-fuzz (фуззинг) и mcp-reality-check (проверка ответов). Каждый инструмент обнаружил собственную ошибку при тестировании реальных серверов, что привело к исправлениям в исходном коде и публикации результатов в публичном лидерборде.

Задача

MCP servers often have undocumented features, unsafe failures, or misleading responses, making it hard to assess their quality and security.

Результат

All three tools found bugs in their own logic during initial real-world testing, leading to upstream fixes and a public leaderboard of MCP server quality. The strongest finding was that 27 tools in the Ant Design MCP server crashed on malformed input, prompting a fix that was already present but unpublished.

Рабочий процесс

• 1. Build a CLI that statically analyzes server source for documentation gaps, hardcoded secrets, and missing tests (mcp-doctor).
• 2. Build a CLI that launches the server via stdio, sends schema-derived malformed inputs to read-only tools, and checks for structured error responses versus crashes (mcp-fuzz).
• 3. Build a CLI that sends realistic inputs, verifies responses against declared output schemas, and flags disguised refusals or empty content (mcp-reality-check).
• 4. Run each tool against popular MCP servers, identify bugs in the tools themselves, fix them, and re-test.
• 5. Report findings to upstream projects via PRs or issues, providing reproducible JSON-RPC payloads when necessary.
• 6. Publish a leaderboard and documentation for community use.

Доказательства

mcp-doctor, against homeassistant-ai/ha-mcp (4.5k stars): the secret scanner was flagging test fixtures and identifier-style constants as hardcoded credentials, and tool detection was counting mock functions inside test files as real tools.

Источник

Running mcp-fuzz against antvis/mcp-server-chart (4.3k stars, official Ant Design MCP server, 27 chart-generation tools) found that all 27 tools crash — raw JSON-RPC -32603 internal errors, not structured tool-level errors — on missing-required or wrong-type input.

Источник