Stopping a crawl when the data stops looking real

说明与用途

自动化方案，使 Scrapy 爬虫在数据质量下降时自动停止，使用 Jev AI 模型对字段进行可行性评分。

问题

Scrapers silently break when selectors change, returning wrong or missing data without raising errors, leading to downstream failures.

结果

The automation recipe enables a Scrapy crawler to self-terminate when data quality drops, catching selector breakage early and saving downstream costs.

工作流

• 1. Install scrapy-jev via uv add scrapy-jev.
• 2. Configure settings: add the addon, specify JEV_FIELDS, set API key, and optional thresholds.
• 3. Run the crawl; JevClient sends each item’s fields to the Jev model for plausibility scoring.
• 4. QualityGatePipeline samples items, checks per-field scores against JEV_FIELD_THRESHOLD, and calculates pass rate.
• 5. If pass rate < JEV_PASS_RATE_THRESHOLD, pipeline triggers crawler.close_spider_async to stop the crawl with a clear reason.

证据

I built scrapy-jev, a Scrapy pipeline that asks a fast, cheap AI model whether each scraped field still looks like a real product name, price, SKU, or category, and stops the crawl when too many items fail the check.

来源

The gate stopped the crawl when the sku selector returned the product title instead of a SKU.

来源

The fix was to stop asking politely and instead tell the engine directly: crawler.engine.close_spider_async(reason).

来源