Building a Resilient, Claude-Powered Web Scraper with YAML (Self-Hosted)

Description and use

HYAE is a self-hosted, YAML-driven scraper powered by Claude AI that adapts to layout changes, reducing maintenance and keeping costs low.

Problem

Traditional web scrapers break when site layouts or class names change, forcing developers to spend time fixing selectors; cloud-based AI scrapers are costly and capped.

Outcome

HYAE provides a resilient, YAML-driven scraping workflow that adapts to site changes without manual selector updates, while remaining self-hosted and cost-effective.

Workflow

• Define a YAML schema with desired output fields
• Run HYAE to fetch pages respecting robots.txt and rate limits
• Claude parses raw HTML and outputs structured JSON or CSV

Evidence

If you've maintained web scrapers at scale, you know the routine: a site updates its layout, class names change, and your XPath or CSS selectors instantly break.

Source

HYAE is a self-hosted, YAML-driven scraping engine powered by Claude AI.

Source

Define a simple YAML file specifying the fields you want to extract (e.g., product titles, prices, specs).

Source

You only pay for the exact tokens you use without recurring platform markups.

Source