I Gave My AI Agents Their Own Documentation Crawler, and Pulled 60 Pages of Clean Markdown in 49 Seconds

Описание и применение

Автоматический инструмент pulpie-mcp позволяет ИИ-агентам скачивать и сохранять полные веб‑документы в чистый Markdown за 49 секунд, избавляя от ручного скачивания и сокращения контента.

Задача

AI agents often lack a reliable way to fetch and store full web documentation as clean Markdown, leading to manual steps and summarized or incomplete content.

Результат

In under 49 seconds, Claude crawled 60 documentation pages, saving them as clean Markdown files, with a ~10s cold start and ~37s crawl time, enabling agents to reference full docs without context window overload.

Рабочий процесс

• 1. Install pulpie-mcp via uv or pip.
• 2. Add the MCP server to Claude or other clients.
• 3. Use tools like fetch_markdown, save_markdown, crawl_docs, or list_library to retrieve or manage docs.
• 4. Agents can request a full docs crawl, which saves files locally and returns metadata.
• 5. Agents can then read or reference specific Markdown files as needed.

Доказательства

I had Claude test crawl_docs against a documentation site with 60 pages.

Источник

The backend cold start was ~10s and crawling 60 pages took ~37s, total ~49s.

Источник

pulpie-mcp uses feyninc/pulpie-orange-small, a 210M parameter encoder model, using ~420 MB VRAM.

Источник

The MCP server starts the backend automatically and shares it across agents, shutting down after 30 minutes of inactivity.

Источник