A web crawler for AI agents with no LLM in the crawl loop

说明与用途

Jevcrawl 是一个开源网页爬虫，使用轻量级 Jev 模型评估页面相关性，使 AI 代理能够快速找到隐藏在几次点击深处的答案，成本低、速度快。

问题

AI agents need answers buried a few clicks deep, but existing approaches are either slow and token‑heavy or expensive due to repeated LLM calls.

结果

On a Python docs example, jevcrawl fetched 6 pages, reduced 80,000 tokens to 1,900, used 11 scoring calls, cost $0.007, and finished in 8 s. In WebWalkerQA benchmark, it achieved 40% correct answers, 54% answer‑page discovery, $0.025 cost, and 27.5 s time per task, outperforming Claude link‑picking and breadth‑first crawlers in cost and speed.

工作流

• 1. Provide a question and a start URL to jevcrawl.
• 2. jevcrawl fetches pages, sending each text block and link to the Jev model for relevance scoring.
• 3. High‑scoring passages are kept; links are prioritized and queued.
• 4. jevcrawl checks if kept passages already answer the goal and stops early if so.
• 5. Returns the answer passages verbatim with source URLs.

证据

I wanted the accuracy of the second option at a price closer to the first. So I built jevcrawl, an open‑source (Apache‑2.0) MCP server and CLI where a small scoring model drives the crawl and no LLM sits inside the crawl loop.

来源

On that example it fetched 6 pages and cut about 80,000 tokens of page text down to about 1,900 tokens of answers. That took 11 scoring calls, cost about $0.007, and finished in 8 seconds.

来源

I used WebWalkerQA (Wu et al., Alibaba Tongyi Lab, ACL 2025), a benchmark of questions buried several clicks deep in real websites. I re‑verified 50 tasks against the live sites and did not look at them during development.

来源