Web scraping workflows often struggle with data consistency, especially when passing context between different stages of a crawl. Crawlee v3.18 addresses this by introducing type-safe routers and optional runtime schema validation. These features allow developers to define strict data contracts for their scraping logic, catching errors at compile time or immediately upon request insertion, rather than deep within execution handlers.
Context and practical value
The source details the release of Crawlee v3.18, highlighting type-safe routers, runtime schema validation for userData, Puppeteer 25 support, and various bug fixes. It also covers v3.17 features like dynamic memory snapshots and custom load signals for autoscaling.
AtlasRepo structures the release notes into actionable sections for developers, emphasizing the practical benefits of type safety and schema validation in reducing runtime errors. It provides concrete steps for implementation and highlights the impact of bug fixes on crawler stability.
Key takeaways
- Type-safe routers allow defining specific userData shapes per label, catching typos and mismatches at compile time.
- Runtime schema validation using libraries like Zod ensures data integrity before handlers execute, failing fast on invalid inputs.
- Puppeteer 25 is now supported, with backward compatibility maintained for versions down to 21.
- Memory autoscaling now dynamically adjusts to container resizing, preventing under-utilization or crashes.
- Custom load signals enable developers to integrate external metrics like proxy health into autoscaling decisions.
- Bug fixes address issues with off-domain redirects, sitemap filtering, and backpressure deadlocks.
Type-Safe Router Labels
Previously, request.userData in Crawlee was a loosely typed dictionary, meaning handlers received the same shape regardless of their registered label. This often required manual casting or narrowing within each handler. With v3.18, developers can declare a route map that defines a specific userData shape for each label. This map is passed as a type argument to the router factory. Consequently, each handler receives request.userData typed according to its specific label. Unknown labels result in compile-time errors, ensuring that routing logic is strictly enforced without any runtime performance cost. This feature is fully backward compatible; omitting the route map retains the original loose typing behavior.
Runtime Schema Validation
While type-safe routers provide compile-time guarantees, they do not protect against runtime data corruption from untyped sources like actor inputs or scraped values. To address this, router factories now accept a schema per label using Standard Schema compliant libraries such as Zod, Valibot, or ArkType. This schema drives both the inferred userData type and its runtime validation. When a request is handled, its userData is validated and coerced against the registered schema before the handler runs. Validation also occurs when requests are added via crawler.addRequests() or enqueueLinks, causing invalid shapes to fail immediately with a non-retryable RequestValidationError. This approach ensures that bad data is rejected at the entry point rather than causing failures deep within the processing pipeline.
Autoscaling and Memory Management Improvements
Crawlee v3.17, covered in this release, introduced dynamic memory snapshots. Previously, the Snapshotter measured available memory once at startup, which could lead to inefficiencies if the container's memory limit changed during execution. Now, if no fixed memoryMbytes is configured, the snapshotter follows the total memory reported by the event manager, allowing the crawler to scale dynamically with container resizing. Additionally, autoscaling internals were refactored around a new LoadSignal interface. Developers can now implement custom load signals to react to overload conditions not natively tracked by Crawlee, such as navigation timeouts or proxy health metrics, providing finer control over resource utilization.
Bug Fixes and Stability Enhancements
Several critical bugs were addressed in this release. The enqueueLinks function now maintains same-domain filtering anchored to the original request, preventing crawlers from wandering off after off-domain redirects. URLs discovered from sitemaps are now correctly filtered by the configured enqueue strategy. The addRequestsBatched function no longer re-submits already enqueued requests, and retries for unprocessed requests are capped to prevent infinite loops. Furthermore, a deadlock issue caused by backpressured sitemap loads during persistState events has been resolved, and final crawler statistics are now persisted only once to reduce overhead.
Practical next steps
- Define a route map interface for your crawler to enforce type safety on userData across different scraping stages.
- Integrate a schema library like Zod to validate userData at runtime, ensuring data integrity before it reaches your handlers.
- Update your Puppeteer dependency to version 25 to leverage the latest browser automation features and security patches.
- Implement custom load signals if your scraping infrastructure relies on external metrics like proxy health or API rate limits.
Limits and verification
- Schema validation is opt-in and requires additional setup with external libraries like Zod, which may increase initial development complexity.
- Custom load signals require manual implementation of the LoadSignal interface, which may not be straightforward for all users.
FAQ
Is the type-safe router feature backward compatible?
Yes, omitting the route map retains the original loose typing behavior, and passing a plain userData shape still works as before.
What happens if userData fails schema validation?
A non-retryable RequestValidationError is thrown, ensuring that invalid data is rejected immediately rather than causing failures later in the process.
Can I use custom load signals for autoscaling?
Yes, you can implement the LoadSignal interface and pass extra signals to the AutoscaledPool to react to custom overload conditions.