Why crawler reliability lives outside the scraper

说明与用途

文章说明了 Enterprise Crawler Framework (ECF) 如何将爬虫逻辑与可靠性基础设施分离，提供对重试、任务所有权和生命周期的显式控制，从而简化了构建稳健爬虫的过程。

问题

Building reliable crawlers is hard because reliability concerns (retries, failures, ownership, lifecycle) are often mixed with scraping logic, leading to brittle systems.

结果

By decoupling reliability concerns from scraping logic, ECF reduces duplicated code, clarifies ownership of retries and failures, and provides optional persistence and queueing, leading to more robust crawler deployments.

工作流

• 1. Define a bot class inheriting from BaseBot and implement execute() with source logic.
• 2. Instantiate the bot with a name and wrap it in a Crawler instance.
• 3. Call crawler.run() to start the crawler; the framework handles retries, leases, and queueing.
• 4. The framework manages lifecycle events (init, cancel, finalize) and optional persistence.
• 5. On failure, the framework applies retry policies, schedules retries, and moves messages to a dead‑letter queue if needed.

证据

The framework does not know what website, API, feed, document source, or business domain the bot belongs to.

来源

ECF keeps HTTP retry concerns separate from event and worker retry policy.

来源