Stopping a crawl when the data stops looking real

Описание и применение

Рецепт автоматизации, позволяющий Scrapy‑краулеру автоматически завершать работу, когда данные перестают выглядеть реальными, используя быстрый AI‑модель Jev для проверки полей.

Задача

Scrapers silently break when selectors change, returning wrong or missing data without raising errors, leading to downstream failures.

Результат

The automation recipe enables a Scrapy crawler to self-terminate when data quality drops, catching selector breakage early and saving downstream costs.

Рабочий процесс

• 1. Install scrapy-jev via uv add scrapy-jev.
• 2. Configure settings: add the addon, specify JEV_FIELDS, set API key, and optional thresholds.
• 3. Run the crawl; JevClient sends each item’s fields to the Jev model for plausibility scoring.
• 4. QualityGatePipeline samples items, checks per-field scores against JEV_FIELD_THRESHOLD, and calculates pass rate.
• 5. If pass rate < JEV_PASS_RATE_THRESHOLD, pipeline triggers crawler.close_spider_async to stop the crawl with a clear reason.

Доказательства

I built scrapy-jev, a Scrapy pipeline that asks a fast, cheap AI model whether each scraped field still looks like a real product name, price, SKU, or category, and stops the crawl when too many items fail the check.

Источник

The gate stopped the crawl when the sku selector returned the product title instead of a SKU.

Источник

The fix was to stop asking politely and instead tell the engine directly: crawler.engine.close_spider_async(reason).

Источник