A web crawler for AI agents with no LLM in the crawl loop

Description and use

Jevcrawl is an open‑source web crawler that uses a lightweight Jev model to score page relevance, enabling AI agents to quickly find answers buried a few clicks deep with low cost and high speed.

Problem

AI agents need answers buried a few clicks deep, but existing approaches are either slow and token‑heavy or expensive due to repeated LLM calls.

Outcome

On a Python docs example, jevcrawl fetched 6 pages, reduced 80,000 tokens to 1,900, used 11 scoring calls, cost $0.007, and finished in 8 s. In WebWalkerQA benchmark, it achieved 40% correct answers, 54% answer‑page discovery, $0.025 cost, and 27.5 s time per task, outperforming Claude link‑picking and breadth‑first crawlers in cost and speed.

Workflow

• 1. Provide a question and a start URL to jevcrawl.
• 2. jevcrawl fetches pages, sending each text block and link to the Jev model for relevance scoring.
• 3. High‑scoring passages are kept; links are prioritized and queued.
• 4. jevcrawl checks if kept passages already answer the goal and stops early if so.
• 5. Returns the answer passages verbatim with source URLs.

Evidence

I wanted the accuracy of the second option at a price closer to the first. So I built jevcrawl, an open‑source (Apache‑2.0) MCP server and CLI where a small scoring model drives the crawl and no LLM sits inside the crawl loop.

Source

On that example it fetched 6 pages and cut about 80,000 tokens of page text down to about 1,900 tokens of answers. That took 11 scoring calls, cost about $0.007, and finished in 8 seconds.

Source

I used WebWalkerQA (Wu et al., Alibaba Tongyi Lab, ACL 2025), a benchmark of questions buried several clicks deep in real websites. I re‑verified 50 tasks against the live sites and did not look at them during development.

Source