Why crawler reliability lives outside the scraper

Описание и применение

В статье описывается, как Enterprise Crawler Framework (ECF) отделяет логику сканирования от инфраструктуры надёжности, обеспечивая явное управление повторными попытками, владением задачами и жизненным циклом, что упрощает разработку устойчивых краулеров.

Задача

Building reliable crawlers is hard because reliability concerns (retries, failures, ownership, lifecycle) are often mixed with scraping logic, leading to brittle systems.

Результат

By decoupling reliability concerns from scraping logic, ECF reduces duplicated code, clarifies ownership of retries and failures, and provides optional persistence and queueing, leading to more robust crawler deployments.

Рабочий процесс

• 1. Define a bot class inheriting from BaseBot and implement execute() with source logic.
• 2. Instantiate the bot with a name and wrap it in a Crawler instance.
• 3. Call crawler.run() to start the crawler; the framework handles retries, leases, and queueing.
• 4. The framework manages lifecycle events (init, cancel, finalize) and optional persistence.
• 5. On failure, the framework applies retry policies, schedules retries, and moves messages to a dead‑letter queue if needed.

Доказательства

The framework does not know what website, API, feed, document source, or business domain the bot belongs to.

Источник

ECF keeps HTTP retry concerns separate from event and worker retry policy.

Источник