Scraping websites and bot-protection

Описание и применение

Автор успешно обошёл защиту от ботов на сайтах генеалогии, используя автоматизацию браузера Playwright, постоянные профили и реальные Chrome‑бинарные файлы.

Задача

Accessing genealogical and public record data was blocked by varying levels of bot protection on target websites.

Результат

The author achieved reliable data extraction from protected genealogy sites by adapting the scraping strategy to the protection level, ultimately relying on headed browser automation and persistent profiles.

Рабочий процесс

• 1. Identify protection level (none, TLS fingerprint, JavaScript, headless detection).
• 2. For no protection, use curl with User-Agent, Referer, and cookie handling.
• 3. If TLS fingerprinting, use curl_cffi or a browser with matching fingerprint.
• 4. For JavaScript challenges (Anubis, Cloudflare JS detection), launch a headed browser via Playwright.
• 5. If headless browsers are blocked, switch to headed mode or use a real Chrome binary.
• 6. Maintain a persistent browser profile to reuse clearance cookies (cf_clearance).
• 7. Automate navigation and data extraction with Playwright scripts.
• 8. Verify data retrieval and handle any remaining challenges manually if needed.

Доказательства

The base level has little to no protection. Some sites do a surface check, but it's not serious protection.

Источник

To bypass any of them, or similar bot-protection mechanisms, just use a browser. You can steer it with browser-automation software, such as Playwright.

Источник

Some sites needed two more things: A persistent browser profile and A real branded Chrome.

Источник