We installed 12,915 MCP servers and started each one. 41% never answered
Описание и применение
Из 12 915 серверов MCP 59 % ответили на протокол, 41 % не ответили. Были выявлены причины отказов и сетевые взаимодействия.
Задача
Most MCP server descriptions claim safety, but it is unclear how they behave when actually installed and started.
Результат
Out of 12,915 servers, 7,656 (59%) answered MCP protocol calls; 1,239 started only after receiving a placeholder key or argument; 5,259 never answered. Reasons for failures included broken packages, key/setting requirements, usage text, HTTP servers, missing dependencies, and SDK incompatibilities.
Рабочий процесс
- 1. Crawl npm and PyPI registries for MCP servers (47,586 packages, run up to 8,000 per crawl).
- 2. Install each package fresh: npm with --ignore-scripts, PyPI wheels only.
- 3. Run each server in gVisor (runsc) sandbox: unprivileged user, read‑only root, 512 MB RAM, 1 CPU, 360 s install, 150 s start.
- 4. Block outbound DNS; all names resolve to traps that record traffic and forward nothing.
- 5. Place decoy credential files in HOME and working directory.
- 6. Observe server via MCP protocol: send initialize, tools/list, then up to 15 tools with canary arguments.
- 7. If server exits with key/setting error, restart with placeholder key or start argument.
- 8. Record stdout, stderr, exit code, and gVisor syscall trace.
- 9. Classify each run as answered, started with placeholder, or never answered.
- 10. Analyze exit messages to categorize reasons for non‑answering.
- 11. Re‑run servers that depend on mcp<2 to test SDK compatibility.
- 12. Aggregate statistics and produce visual summaries.
Доказательства
7,656 of 12,915 answered initialize and tools/list: 64% on npm, 52% on PyPI.
5,259 never answered. We read what each package printed as it exited, last run first.
2,239 ran npm install scripts; 1,484 reached for the network at startup; 295 opened a decoy credential file; 73 runs sent the placeholder key over the network.
- Дата записи: 2026-10-14T06:41:17.341Z