Why Wellfound Scraper Runs Drop Items After 300 Seconds

Description and use

Switching from synchronous to asynchronous execution of the Wellfound Scraper avoids data loss due to the 5‑minute timeout, ensuring complete job listings are captured.

Problem

When using Apify’s Wellfound Jobs Scraper synchronously, the 5‑minute timeout causes partial data loss, dropping items after 300 seconds.

Outcome

Switching from synchronous to asynchronous execution of the Wellfound scraper prevents data loss due to the 5‑minute timeout, ensuring complete job listings are captured for downstream analysis.

Workflow

• 1. Prepare ApifyClient with API token
• 2. Define actor input (startUrls, filters, maxItems)
• 3. For short runs, call actor synchronously and wait for completion
• 4. For long runs, call actor asynchronously (wait_for_finish=False) and capture run ID
• 5. Poll run status or set up a webhook to receive completion notification
• 6. Once run succeeds, fetch dataset items from Apify and process downstream

Evidence

The Apify synchronous run endpoint imposes a hard limit of 300 seconds (5 minutes) before timing out and returning an HTTP 408 error.

Source

For longer runs, you must switch to an asynchronous approach. This involves initiating the run and then either polling the run status periodically or configuring a webhook to notify your downstream system when the run completes.

Source