Building a Gradcracker scraper with Crawlee: anti-bot failures and redirect bugs

Описание и применение

Скрипт для Gradcracker, использующий Crawlee и Camoufox, успешно обходит анти‑бот защиту и переменные перенаправления, но остаётся медленным из‑за ограничений сайта.

Задача

Gradcracker’s aggressive bot detection and unpredictable redirect behavior made scraping job listings unreliable and slow.

Результат

Scraper now consistently retrieves job listings from Gradcracker, handling bot detection and variable redirects, though it remains slower than other sources.

Рабочий процесс

• Generate start URLs for each location/role
• Queue list pages labeled "gradcracker-list-page"
• Router enqueues individual job pages
• Scrape job page content and locate "Apply Online" button
• Click button and poll new tab URL until stable and not on Gradcracker
• Scrape application page
• Return to next list page and repeat

Доказательства

The fix was swapping in Camoufox, a Firefox-based browser built specifically for anti-detection.

Источник

Stop trying to predict when navigation is done. Instead, just poll the URL every 100ms and wait until it hasn't changed for three consecutive checks.

Источник