Scraping websites and bot-protection

Description and use

The author successfully bypassed bot protection on genealogy sites using Playwright browser automation, persistent profiles, and real Chrome binaries.

Problem

Accessing genealogical and public record data was blocked by varying levels of bot protection on target websites.

Outcome

The author achieved reliable data extraction from protected genealogy sites by adapting the scraping strategy to the protection level, ultimately relying on headed browser automation and persistent profiles.

Workflow

• 1. Identify protection level (none, TLS fingerprint, JavaScript, headless detection).
• 2. For no protection, use curl with User-Agent, Referer, and cookie handling.
• 3. If TLS fingerprinting, use curl_cffi or a browser with matching fingerprint.
• 4. For JavaScript challenges (Anubis, Cloudflare JS detection), launch a headed browser via Playwright.
• 5. If headless browsers are blocked, switch to headed mode or use a real Chrome binary.
• 6. Maintain a persistent browser profile to reuse clearance cookies (cf_clearance).
• 7. Automate navigation and data extraction with Playwright scripts.
• 8. Verify data retrieval and handle any remaining challenges manually if needed.

Evidence

The base level has little to no protection. Some sites do a surface check, but it's not serious protection.

Source

To bypass any of them, or similar bot-protection mechanisms, just use a browser. You can steer it with browser-automation software, such as Playwright.

Source

Some sites needed two more things: A persistent browser profile and A real branded Chrome.

Source