Building a Gradcracker scraper with Crawlee: anti-bot failures and redirect bugs

Description and use

A Gradcracker scraper built with Crawlee and Camoufox overcomes bot detection and variable redirects, achieving reliable data extraction at the cost of speed due to site constraints.

Problem

Gradcracker’s aggressive bot detection and unpredictable redirect behavior made scraping job listings unreliable and slow.

Outcome

Scraper now consistently retrieves job listings from Gradcracker, handling bot detection and variable redirects, though it remains slower than other sources.

Workflow

• Generate start URLs for each location/role
• Queue list pages labeled "gradcracker-list-page"
• Router enqueues individual job pages
• Scrape job page content and locate "Apply Online" button
• Click button and poll new tab URL until stable and not on Gradcracker
• Scrape application page
• Return to next list page and repeat

Evidence

The fix was swapping in Camoufox, a Firefox-based browser built specifically for anti-detection.

Source

Stop trying to predict when navigation is done. Instead, just poll the URL every 100ms and wait until it hasn't changed for three consecutive checks.

Source