Why Wellfound Scraper Runs Drop Items After 300 Seconds

Описание и применение

Переключение с синхронного запуска Wellfound Scraper на асинхронный позволяет избежать потери данных из‑за 5‑минутного тайм‑аута, обеспечивая полное получение вакансий.

Задача

When using Apify’s Wellfound Jobs Scraper synchronously, the 5‑minute timeout causes partial data loss, dropping items after 300 seconds.

Результат

Switching from synchronous to asynchronous execution of the Wellfound scraper prevents data loss due to the 5‑minute timeout, ensuring complete job listings are captured for downstream analysis.

Рабочий процесс

• 1. Prepare ApifyClient with API token
• 2. Define actor input (startUrls, filters, maxItems)
• 3. For short runs, call actor synchronously and wait for completion
• 4. For long runs, call actor asynchronously (wait_for_finish=False) and capture run ID
• 5. Poll run status or set up a webhook to receive completion notification
• 6. Once run succeeds, fetch dataset items from Apify and process downstream

Доказательства

The Apify synchronous run endpoint imposes a hard limit of 300 seconds (5 minutes) before timing out and returning an HTTP 408 error.

Источник

For longer runs, you must switch to an asynchronous approach. This involves initiating the run and then either polling the run status periodically or configuring a webhook to notify your downstream system when the run completes.

Источник