A web crawler for AI agents with no LLM in the crawl loop

Описание и применение

Jevcrawl – это открытый веб‑краулер, использующий небольшую модель Jev для оценки релевантности страниц, позволяющий AI‑агенту быстро находить ответы, скрытые за несколькими кликами, при низкой стоимости и высокой скорости.

Задача

AI agents need answers buried a few clicks deep, but existing approaches are either slow and token‑heavy or expensive due to repeated LLM calls.

Результат

On a Python docs example, jevcrawl fetched 6 pages, reduced 80,000 tokens to 1,900, used 11 scoring calls, cost $0.007, and finished in 8 s. In WebWalkerQA benchmark, it achieved 40% correct answers, 54% answer‑page discovery, $0.025 cost, and 27.5 s time per task, outperforming Claude link‑picking and breadth‑first crawlers in cost and speed.

Рабочий процесс

• 1. Provide a question and a start URL to jevcrawl.
• 2. jevcrawl fetches pages, sending each text block and link to the Jev model for relevance scoring.
• 3. High‑scoring passages are kept; links are prioritized and queued.
• 4. jevcrawl checks if kept passages already answer the goal and stops early if so.
• 5. Returns the answer passages verbatim with source URLs.

Доказательства

I wanted the accuracy of the second option at a price closer to the first. So I built jevcrawl, an open‑source (Apache‑2.0) MCP server and CLI where a small scoring model drives the crawl and no LLM sits inside the crawl loop.

Источник

On that example it fetched 6 pages and cut about 80,000 tokens of page text down to about 1,900 tokens of answers. That took 11 scoring calls, cost about $0.007, and finished in 8 seconds.

Источник

I used WebWalkerQA (Wu et al., Alibaba Tongyi Lab, ACL 2025), a benchmark of questions buried several clicks deep in real websites. I re‑verified 50 tasks against the live sites and did not look at them during development.

Источник