I crawled the Tranco Top 100k with a Rust engine that never passed 188 MB of RAM

Description and use

PageSift is a single‑binary Rust web scraper compatible with Firecrawl, using only 188 MB RAM to crawl Tranco’s 100k domains, outperforming Spider in memory and extraction quality.

Problem

Web scraping for an AI product was costly, resource‑heavy, and often captured captcha or interstitial pages as content.

Outcome

PageSift maintained a flat 188.4 MB peak RSS while crawling 100k domains, produced 60,659 valid Markdown docs (96.38% conversion), and avoided memory leaks, outperforming Spider’s 8,978.9 MB RSS and 82.82% conversion.

Workflow

• 1. Build PageSift (v0.50.0) from source or pull Docker image. 2. Run with API key and expose port 3002. 3. Use Firecrawl‑compatible API to scrape URLs. 4. PageSift streams clean Markdown, returns ACCESS_BARRIER on blocked sites.
• 5. Compare performance against Spider on identical hardware and settings (768 async concurrency, 3.5 s connect / 10 s total timeout, Blake3 dedup).
• 6. Evaluate metrics: peak RSS, valid Markdown docs, extraction conversion, memory behavior.

Evidence

I built PageSift, a single-binary web scraping engine in Rust. It’s Firecrawl v1/v2 API-compatible, turns pages into clean Markdown for LLM pipelines, and just hit v0.50.0.

Source

On a live crawl of the Tranco Top 100,000 domains, its peak memory stayed flat at 188.4 MB while Spider (also Rust) ballooned to ~9.0 GB and got OOM‑killed.

Source

The full dataset and scripts: bench/results/final/BENCHMARK_100K_PAGESIFT_VS_SPIDER.md

Source