I built three tools to audit MCP servers. Each one found a bug in itself first.

Description and use

Three CLI tools—mcp-doctor, mcp-fuzz, and mcp-reality-check—were built to audit MCP servers for documentation, safe failure, and honest responses. Each tool found bugs in itself during real-world testing, leading to upstream fixes and a public leaderboard of MCP server quality.

Problem

MCP servers often have undocumented features, unsafe failures, or misleading responses, making it hard to assess their quality and security.

Outcome

All three tools found bugs in their own logic during initial real-world testing, leading to upstream fixes and a public leaderboard of MCP server quality. The strongest finding was that 27 tools in the Ant Design MCP server crashed on malformed input, prompting a fix that was already present but unpublished.

Workflow

• 1. Build a CLI that statically analyzes server source for documentation gaps, hardcoded secrets, and missing tests (mcp-doctor).
• 2. Build a CLI that launches the server via stdio, sends schema-derived malformed inputs to read-only tools, and checks for structured error responses versus crashes (mcp-fuzz).
• 3. Build a CLI that sends realistic inputs, verifies responses against declared output schemas, and flags disguised refusals or empty content (mcp-reality-check).
• 4. Run each tool against popular MCP servers, identify bugs in the tools themselves, fix them, and re-test.
• 5. Report findings to upstream projects via PRs or issues, providing reproducible JSON-RPC payloads when necessary.
• 6. Publish a leaderboard and documentation for community use.

Evidence

mcp-doctor, against homeassistant-ai/ha-mcp (4.5k stars): the secret scanner was flagging test fixtures and identifier-style constants as hardcoded credentials, and tool detection was counting mock functions inside test files as real tools.

Source

Running mcp-fuzz against antvis/mcp-server-chart (4.3k stars, official Ant Design MCP server, 27 chart-generation tools) found that all 27 tools crash — raw JSON-RPC -32603 internal errors, not structured tool-level errors — on missing-required or wrong-type input.

Source