We installed 12,915 MCP servers and started each one. 41% never answered

Description and use

Of 12,915 MCP servers, 59 % answered the protocol, 41 % did not. Causes of failures and network interactions were identified.

Problem

Most MCP server descriptions claim safety, but it is unclear how they behave when actually installed and started.

Outcome

Out of 12,915 servers, 7,656 (59%) answered MCP protocol calls; 1,239 started only after receiving a placeholder key or argument; 5,259 never answered. Reasons for failures included broken packages, key/setting requirements, usage text, HTTP servers, missing dependencies, and SDK incompatibilities.

Workflow

• 1. Crawl npm and PyPI registries for MCP servers (47,586 packages, run up to 8,000 per crawl).
• 2. Install each package fresh: npm with --ignore-scripts, PyPI wheels only.
• 3. Run each server in gVisor (runsc) sandbox: unprivileged user, read‑only root, 512 MB RAM, 1 CPU, 360 s install, 150 s start.
• 4. Block outbound DNS; all names resolve to traps that record traffic and forward nothing.
• 5. Place decoy credential files in HOME and working directory.
• 6. Observe server via MCP protocol: send initialize, tools/list, then up to 15 tools with canary arguments.
• 7. If server exits with key/setting error, restart with placeholder key or start argument.
• 8. Record stdout, stderr, exit code, and gVisor syscall trace.
• 9. Classify each run as answered, started with placeholder, or never answered.
• 10. Analyze exit messages to categorize reasons for non‑answering.
• 11. Re‑run servers that depend on mcp<2 to test SDK compatibility.
• 12. Aggregate statistics and produce visual summaries.

Evidence

7,656 of 12,915 answered initialize and tools/list: 64% on npm, 52% on PyPI.

Source

5,259 never answered. We read what each package printed as it exited, last run first.

Source

2,239 ran npm install scripts; 1,484 reached for the network at startup; 295 opened a decoy credential file; 73 runs sent the placeholder key over the network.

Source