Stop Rewriting Your Web Scraper for Every New Site

Description and use

Switching from copying scripts to a config‑driven engine lets you add new sites quickly without code changes, reducing errors and improving data quality.

Problem

Maintaining multiple near-identical web‑scraping scripts for each new site leads to duplicated effort, fragile code, and data quality issues.

Outcome

A single, config‑driven scraper engine replaces dozens of hard‑coded scripts, reducing maintenance effort, improving data quality, and enabling unattended runs with retries and deduplication.

Workflow

• 1. Write a generic scraper engine that reads a YAML config.
• 2. Create a YAML file for each target site with start URLs, item container selector, field selectors and types, pagination style, rate limits, and dedup keys.
• 3. Run the engine against the config to fetch, parse, validate, deduplicate, and export data.
• 4. Monitor logs for dropped rows and retry failures; adjust config as needed.

Evidence

Every web scraper I've ever written started the same way: a quick script for one site, hardcoded selectors, done in an afternoon.

Source

Selectors live in code. A new site means editing Python, re-testing, and redeploying — for what is really just data about the site.

Source

The pattern I use in production flips the script inside out. Everything site‑specific — start URLs, the item container, per‑field selectors and types, pagination style, rate limits, dedup keys — moves into a YAML config.

Source