apify/crawlee — Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers.

Description and use

**Brief Summary (2‑3 sentences)**
Crawlee is an open‑source Node.js library (JavaScript/TypeScript) that unifies web‑scraping, browser automation, and file downloading across Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. It lets developers build reliable crawlers that can harvest HTML, PDFs, images, and other assets in both headful and headless modes, with built‑in proxy rotation and strong TypeScript typings. With 23 k+ GitHub stars and active maintenance, it’s a mature foundation for feeding AI/LLM pipelines, RAG systems, and knowledge‑base indexing.

---

Value Proposition

• **Turn web content into searchable knowledge:** Crawlee can systematically crawl internal portals, public sites, or document repositories and dump the results (HTML, PDFs, images, JSON) into vector stores or databases that LLM‑powered assistants can query.
• **Unified API across multiple back‑ends:** Whether you need a fast Cheerio‑style DOM parser, a full‑browser Playwright/Puppeteer session, or simple HTTP requests, the same SDK/CLI surface works, reducing engineering overhead.
• **Production‑grade reliability:** Built‑in retry logic, concurrency control, auto‑throttling, and proxy rotation help avoid bans and keep crawls stable at scale.

Practical Adoption Path

1. **Prototype:** Install the crawlee npm package, pick the appropriate CheerioCrawler, PuppeteerCrawler, or PlaywrightCrawler class, and write a small script that extracts a few pages into JSON.
2. **Integrate with your data pipeline:** Pipe the crawler output to your existing ingestion layer (e.g., write to a document store, push to an embedding service, or dump to S3). The SDK provides hooks (requestHandler, failedRequestHandler, pageFunction) for custom processing.
3. **Scale & secure:** Enable the built‑in proxy pool or configure your own proxy service, adjust concurrency limits, and add a CI job that runs the crawler on a schedule. Use the CLI (crawlee run) for simple automation or embed the SDK in a microservice for on‑demand crawling.
4. **Monitor & iterate:** Leverage the built‑in logging and Dataset/KeyValueStore persistence to track crawl health, then refine selectors or add headful mode for pages that require JavaScript rendering.

Production Readiness

• **Activity & community:** 23 k stars, 1.3 k forks, recent commits (as of 2026‑05‑13), and a vibrant ecosystem of plugins and examples.
• **Maturity:** The library has been battle‑tested in Apify’s own SaaS platform, supporting large‑scale crawling workloads with automated scaling and fault tolerance.
• **Type safety & documentation:** Full TypeScript typings, extensive API docs, and a CLI make onboarding fast for both JS and TS teams.
• **Risk considerations:** No immediate licensing or security red flags, but a final review of the MIT license compliance and any third‑party proxy services is advisable.

Overall, Crawlee is production‑ready for pilots and can be rolled into a full‑scale knowledge‑base ingestion pipeline with minimal friction.

Problem

apify/crawlee helps make internal knowledge searchable and usable by assistants.

Outcome

High for an OSS candidate: recent activity, adoption, and ecosystem signals are strong enough for a serious pilot.

Workflow

• index knowledge bases
• improve search over documents
• ground assistant answers

Evidence

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Source