The release of Crawlee v3.16 marks a significant shift in web scraping architecture by integrating artificial intelligence directly into the crawling pipeline. This update addresses the fragility of traditional CSS selectors by introducing StagehandCrawler, which allows developers to define interactions using natural language. Beyond AI capabilities, the release improves developer experience through native async iteration support for datasets and key-value stores, alongside utilities for automated sitemap discovery and more granular control over anti-bot challenge resolution.
Context and practical value
The source details the release of Crawlee v3.16, highlighting the new StagehandCrawler for AI-powered interactions, async iterator support for data stores, a sitemap discovery utility, and improved Cloudflare challenge handling with configurable callbacks.
This article synthesizes the technical updates into a structured guide, distinguishing between the use cases for AI-driven crawling versus traditional methods, and provides practical implementation steps for developers adopting the new async iteration and sitemap discovery features.
Key takeaways
- StagehandCrawler enables natural language interactions, reducing reliance on brittle CSS selectors.
- Dataset and KeyValueStore now support native async iteration, simplifying data processing loops.
- A new utility automatically discovers valid sitemaps from a list of root URLs.
- Cloudflare challenge handling is now configurable with custom callbacks for detection and interaction.
- AI-powered crawling integrates seamlessly with existing Crawlee infrastructure like proxy rotation and autoscaling.
Natural Language Automation with StagehandCrawler
The headline feature of this release is the integration of Browserbase's Stagehand via the @crawlee/stagehand package. This allows developers to replace complex XPath or CSS selectors with plain English instructions. The enhanced page object provides four primary methods: page.act() for performing actions, page.extract() for retrieving structured data using Zod schemas, page.observe() for discovering available actions, and page.agent() for autonomous multi-step workflows. Because StagehandCrawler extends BrowserCrawler, it inherits standard features such as request queues, session management, and proxy rotation, ensuring that AI-driven automation remains scalable and reliable.
Simplified Data Iteration
Previously, iterating over large datasets or key-value stores required manual pagination logic. Crawlee v3.16 introduces native support for async iterators (for await...of) for both Dataset and KeyValueStore. This change allows developers to iterate over items, keys, or values directly without managing offsets or cursors. The implementation handles pagination internally, making code cleaner and more memory-efficient. Existing methods like listItems() now return hybrid objects that support both traditional awaiting and modern async iteration patterns.
Automated Sitemap Discovery
The new discoverValidSitemaps utility in @crawlee/utils simplifies the seeding process for crawlers. Given a list of URLs, the function automatically checks robots.txt for sitemap declarations and probes common paths such as /sitemap.xml and /sitemap_index.xml. This is particularly useful for large-scale crawling projects where manually identifying sitemap locations for each domain is impractical. The utility returns an async generator, allowing for efficient processing of discovered sitemaps.
Enhanced Cloudflare Challenge Handling
The handleCloudflareChallenge helper has been updated to accept configuration callbacks, providing greater control over how anti-bot challenges are detected and solved. Developers can now customize the click position calculation, override the clicking logic, and adjust detection thresholds for challenge and block pages. Additionally, a preChallengeSleepSecs option allows for configurable delays before interaction attempts. These improvements are essential for environments where default detection mechanisms fail due to specific rendering conditions or network configurations.
Practical next steps
- Install the @crawlee/stagehand and @browserbasehq/stagehand packages to enable AI-powered crawling capabilities.
- Refactor existing data processing loops to use for await...of syntax for cleaner iteration over Datasets and KeyValueStores.
- Implement the discoverValidSitemaps utility to automate the seeding of crawlers for new domains without manual sitemap lookup.
- Configure custom callbacks in handleCloudflareChallenge if default detection fails in your specific deployment environment.
Limits and verification
- AI-powered crawling requires API keys and incurs additional costs compared to traditional selector-based methods.
- StagehandCrawler may be slower than PlaywrightCrawler for sites with stable, well-defined structures.
- Natural language instructions may occasionally misinterpret page elements, requiring careful schema definition for extraction.
FAQ
Is StagehandCrawler a replacement for PlaywrightCrawler?
No, it is a specialized tool for complex or dynamic layouts. For stable sites, PlaywrightCrawler remains faster and more cost-effective as it does not require AI API calls.
Do I need to rewrite existing crawlers to use async iterators?
No, existing code using listItems() or listKeys() continues to work. The new async iterator support is an additive feature for cleaner code.
How does discoverValidSitemaps work?
It checks robots.txt for sitemap declarations and probes common sitemap paths, returning an async generator of valid sitemap URLs.