I built three tools to audit MCP servers. Each one found a bug in itself first.

说明与用途

构建了三款 CLI 工具（mcp-doctor、mcp-fuzz、mcp-reality-check）来审核 MCP 服务器的文档、错误处理和响应真实性。每个工具在真实服务器测试中发现自身缺陷，促成了上游修复并发布了公共排行榜。

问题

MCP servers often have undocumented features, unsafe failures, or misleading responses, making it hard to assess their quality and security.

结果

All three tools found bugs in their own logic during initial real-world testing, leading to upstream fixes and a public leaderboard of MCP server quality. The strongest finding was that 27 tools in the Ant Design MCP server crashed on malformed input, prompting a fix that was already present but unpublished.

工作流

• 1. Build a CLI that statically analyzes server source for documentation gaps, hardcoded secrets, and missing tests (mcp-doctor).
• 2. Build a CLI that launches the server via stdio, sends schema-derived malformed inputs to read-only tools, and checks for structured error responses versus crashes (mcp-fuzz).
• 3. Build a CLI that sends realistic inputs, verifies responses against declared output schemas, and flags disguised refusals or empty content (mcp-reality-check).
• 4. Run each tool against popular MCP servers, identify bugs in the tools themselves, fix them, and re-test.
• 5. Report findings to upstream projects via PRs or issues, providing reproducible JSON-RPC payloads when necessary.
• 6. Publish a leaderboard and documentation for community use.

证据

mcp-doctor, against homeassistant-ai/ha-mcp (4.5k stars): the secret scanner was flagging test fixtures and identifier-style constants as hardcoded credentials, and tool detection was counting mock functions inside test files as real tools.

来源

Running mcp-fuzz against antvis/mcp-server-chart (4.3k stars, official Ant Design MCP server, 27 chart-generation tools) found that all 27 tools crash — raw JSON-RPC -32603 internal errors, not structured tool-level errors — on missing-required or wrong-type input.

来源