I Built Scrapers That Heal Themselves (and Won an NVIDIA DGX Spark)

说明与用途

爬虫在网站重构后经常无声失效。本文描述了一个结合 Bright Data、Port 和 SigNoz 的系统，能够自动检测、修复并审计失效的爬虫，赢得了黑客松比赛。

问题

Scrapers often fail silently after site redesigns, causing data drift that is only discovered weeks later.

结果

The system won first place at the WeMakeDevs Zero Downtime Hackathon and demonstrated reliable, self‑healing scrapers that maintain data integrity.

工作流

• Schedule scrape via Port workflow
• Run Bright Data scraper to extract guide data
• Detect drift using verifier against frozen fixture
• If drift, call Bright Data heal to propose fix
• Verify proposal deterministically
• Approve fix if verification passes
• Update scraper health status in Port
• Publish ranked top 10 shows and movies
• Provide one‑click record button on Philo

证据

Scrapers rot without anyone noticing

来源

Bright Data’s Scraper Studio builds a collector from a natural language description of the data instead of selectors

来源

When drift is detected, the healer calls scraper heal with the collector ID and a plain language symptom description

来源