Scraping websites and bot-protection

说明与用途

作者通过使用 Playwright 浏览器自动化、持久化配置文件和真实 Chrome 二进制文件，成功绕过了家谱网站的机器人防护。

问题

Accessing genealogical and public record data was blocked by varying levels of bot protection on target websites.

结果

The author achieved reliable data extraction from protected genealogy sites by adapting the scraping strategy to the protection level, ultimately relying on headed browser automation and persistent profiles.

工作流

• 1. Identify protection level (none, TLS fingerprint, JavaScript, headless detection).
• 2. For no protection, use curl with User-Agent, Referer, and cookie handling.
• 3. If TLS fingerprinting, use curl_cffi or a browser with matching fingerprint.
• 4. For JavaScript challenges (Anubis, Cloudflare JS detection), launch a headed browser via Playwright.
• 5. If headless browsers are blocked, switch to headed mode or use a real Chrome binary.
• 6. Maintain a persistent browser profile to reuse clearance cookies (cf_clearance).
• 7. Automate navigation and data extraction with Playwright scripts.
• 8. Verify data retrieval and handle any remaining challenges manually if needed.

证据

The base level has little to no protection. Some sites do a surface check, but it's not serious protection.

来源

To bypass any of them, or similar bot-protection mechanisms, just use a browser. You can steer it with browser-automation software, such as Playwright.

来源

Some sites needed two more things: A persistent browser profile and A real branded Chrome.

来源