Stopping a crawl when the data stops looking real

Description and use

Automation recipe enabling a Scrapy crawler to self‑terminate when data quality degrades, using the Jev AI model to plausibly score fields.

Problem

Scrapers silently break when selectors change, returning wrong or missing data without raising errors, leading to downstream failures.

Outcome

The automation recipe enables a Scrapy crawler to self-terminate when data quality drops, catching selector breakage early and saving downstream costs.

Workflow

• 1. Install scrapy-jev via uv add scrapy-jev.
• 2. Configure settings: add the addon, specify JEV_FIELDS, set API key, and optional thresholds.
• 3. Run the crawl; JevClient sends each item’s fields to the Jev model for plausibility scoring.
• 4. QualityGatePipeline samples items, checks per-field scores against JEV_FIELD_THRESHOLD, and calculates pass rate.
• 5. If pass rate < JEV_PASS_RATE_THRESHOLD, pipeline triggers crawler.close_spider_async to stop the crawl with a clear reason.

Evidence

I built scrapy-jev, a Scrapy pipeline that asks a fast, cheap AI model whether each scraped field still looks like a real product name, price, SKU, or category, and stops the crawl when too many items fail the check.

Source

The gate stopped the crawl when the sku selector returned the product title instead of a SKU.

Source

The fix was to stop asking politely and instead tell the engine directly: crawler.engine.close_spider_async(reason).

Source