I Built Scrapers That Heal Themselves (and Won an NVIDIA DGX Spark)

Description and use

Scrapers often fail silently after site redesigns. The article describes a system combining Bright Data, Port, and SigNoz to automatically detect, repair, and audit failing scrapers, winning a hackathon.

Problem

Scrapers often fail silently after site redesigns, causing data drift that is only discovered weeks later.

Outcome

The system won first place at the WeMakeDevs Zero Downtime Hackathon and demonstrated reliable, self‑healing scrapers that maintain data integrity.

Workflow

• Schedule scrape via Port workflow
• Run Bright Data scraper to extract guide data
• Detect drift using verifier against frozen fixture
• If drift, call Bright Data heal to propose fix
• Verify proposal deterministically
• Approve fix if verification passes
• Update scraper health status in Port
• Publish ranked top 10 shows and movies
• Provide one‑click record button on Philo

Evidence

Scrapers rot without anyone noticing

Source

Bright Data’s Scraper Studio builds a collector from a natural language description of the data instead of selectors

Source

When drift is detected, the healer calls scraper heal with the collector ID and a plain language symptom description

Source