I crawled the Tranco Top 100k with a Rust engine that never passed 188 MB of RAM

Описание и применение

PageSift – однобинарный движок на Rust, совместимый с Firecrawl, использует 188 МБ памяти при обходе 100 000 доменов Tranco, превосходя Spider в памяти и качестве извлечения.

Задача

Web scraping for an AI product was costly, resource‑heavy, and often captured captcha or interstitial pages as content.

Результат

PageSift maintained a flat 188.4 MB peak RSS while crawling 100k domains, produced 60,659 valid Markdown docs (96.38% conversion), and avoided memory leaks, outperforming Spider’s 8,978.9 MB RSS and 82.82% conversion.

Рабочий процесс

• 1. Build PageSift (v0.50.0) from source or pull Docker image. 2. Run with API key and expose port 3002. 3. Use Firecrawl‑compatible API to scrape URLs. 4. PageSift streams clean Markdown, returns ACCESS_BARRIER on blocked sites.
• 5. Compare performance against Spider on identical hardware and settings (768 async concurrency, 3.5 s connect / 10 s total timeout, Blake3 dedup).
• 6. Evaluate metrics: peak RSS, valid Markdown docs, extraction conversion, memory behavior.

Доказательства

I built PageSift, a single-binary web scraping engine in Rust. It’s Firecrawl v1/v2 API-compatible, turns pages into clean Markdown for LLM pipelines, and just hit v0.50.0.

Источник

On a live crawl of the Tranco Top 100,000 domains, its peak memory stayed flat at 188.4 MB while Spider (also Rust) ballooned to ~9.0 GB and got OOM‑killed.

Источник

The full dataset and scripts: bench/results/final/BENCHMARK_100K_PAGESIFT_VS_SPIDER.md

Источник