Building a Gradcracker scraper with Crawlee: anti-bot failures and redirect bugs

说明与用途

使用 Crawlee 和 Camoufox 构建的 Gradcracker 抓取器成功绕过了反爬虫保护和可变重定向，能够可靠地提取数据，但由于网站限制导致速度较慢。

问题

Gradcracker’s aggressive bot detection and unpredictable redirect behavior made scraping job listings unreliable and slow.

结果

Scraper now consistently retrieves job listings from Gradcracker, handling bot detection and variable redirects, though it remains slower than other sources.

工作流

• Generate start URLs for each location/role
• Queue list pages labeled "gradcracker-list-page"
• Router enqueues individual job pages
• Scrape job page content and locate "Apply Online" button
• Click button and poll new tab URL until stable and not on Gradcracker
• Scrape application page
• Return to next list page and repeat

证据

The fix was swapping in Camoufox, a Firefox-based browser built specifically for anti-detection.

来源

Stop trying to predict when navigation is done. Instead, just poll the URL every 100ms and wait until it hasn't changed for three consecutive checks.

来源