Why Handling Apify Run Timeouts Prevents Double-Processing

Описание и применение

Обработка тайм‑аутов в Apify позволяет избежать двойной обработки данных, используя асинхронные запуски, вебхуки и внешнее хранилище URL.

Задача

When an Apify Actor run exceeds the 5‑minute synchronous limit, the API returns a 408 error while the run may still be processing in the background, leading to incomplete data and potential duplicate processing when re‑executed.

Результат

The pipeline now reliably handles timeouts, processes only new image URLs, and prevents duplicate downstream actions.

Рабочий процесс

• 1. Initiate Actor run asynchronously via /v2/acts/<actor>/runs
• 2. Use Apify Trigger node in n8n to fire on run completion
• 3. Fetch dataset items from the completed run
• 4. For each image URL, query external state store (DB, Google Sheets, etc.)
• 5. If URL is new, process it (e.g., send to Slack, push to warehouse) and add to state store
• 6. If URL exists, discard it

Доказательства

Apify's synchronous run endpoint has a hard cap of 300 seconds. If a run exceeds this 5-minute limit, it returns an HTTP 408 error.

Источник

The key is to maintain an external record of processed url fields from the Actor's output.

Источник

For event-driven workflows, n8n's Apify Trigger node is ideal.

Источник