I crawled the Tranco Top 100k with a Rust engine that never passed 188 MB of RAM

说明与用途

PageSift 是一个兼容 Firecrawl 的单二进制 Rust 爬虫，使用 188 MB 内存爬取 Tranco 100k 域名，优于 Spider 的内存占用和提取质量。

问题

Web scraping for an AI product was costly, resource‑heavy, and often captured captcha or interstitial pages as content.

结果

PageSift maintained a flat 188.4 MB peak RSS while crawling 100k domains, produced 60,659 valid Markdown docs (96.38% conversion), and avoided memory leaks, outperforming Spider’s 8,978.9 MB RSS and 82.82% conversion.

工作流

• 1. Build PageSift (v0.50.0) from source or pull Docker image. 2. Run with API key and expose port 3002. 3. Use Firecrawl‑compatible API to scrape URLs. 4. PageSift streams clean Markdown, returns ACCESS_BARRIER on blocked sites.
• 5. Compare performance against Spider on identical hardware and settings (768 async concurrency, 3.5 s connect / 10 s total timeout, Blake3 dedup).
• 6. Evaluate metrics: peak RSS, valid Markdown docs, extraction conversion, memory behavior.

证据

I built PageSift, a single-binary web scraping engine in Rust. It’s Firecrawl v1/v2 API-compatible, turns pages into clean Markdown for LLM pipelines, and just hit v0.50.0.

来源

On a live crawl of the Tranco Top 100,000 domains, its peak memory stayed flat at 188.4 MB while Spider (also Rust) ballooned to ~9.0 GB and got OOM‑killed.

来源

The full dataset and scripts: bench/results/final/BENCHMARK_100K_PAGESIFT_VS_SPIDER.md

来源