Why Batching Instagram Profiles Reduces Your Actor Start Costs

Описание и применение

Пошаговый процесс интеграции Apify Instagram Profile Scraper в пайплайн с учётом хранения времени последней выгрузки, чтобы избежать дублирования данных и снизить расходы на запуск актёра.

Задача

Data engineers struggle to efficiently retrieve and process Instagram profile and post data without re‑processing old data or incurring unnecessary costs due to rate limits and session management.

Результат

By tracking scraped_at timestamps and filtering new posts, the workflow prevents duplicate processing, ensures idempotency, and reduces actor usage costs.

Рабочий процесс

• 1. Store the latest scraped_at timestamp for each monitored username in a data store (e.g., database, JSON file).
• 2. Before each Actor run, query the store to retrieve the most recent scraped_at per username.
• 3. Prepare the Actor input: include all usernames (for profile data) and set maxPosts as needed; optionally filter usernames that have new posts based on stored timestamps.
• 4. Call the Instagram Profile Scraper Actor via ApifyClient with the prepared input.
• 5. Retrieve the Actor’s default dataset items after completion.
• 6. Iterate over dataset items: if item contains post_url, compare its pub_date to the stored last_processed_date for that username; if newer, add to new_posts list; otherwise ignore. If item contains author_meta, treat as a profile record and add to updated_profiles list.
• 7. After processing, push new_posts and updated_profiles to downstream systems (e.g., n8n, data warehouse).
• 8. Update the stored scraped_at timestamps with the latest values from the processed items for future runs.

Доказательства

You can avoid re-processing old data by maintaining a record of the last scraped_at timestamp for each profile or post.

Источник

The instagram-profile-scraper Actor generates distinct profile record and post record types, each carrying a scraped_at timestamp in ISO 8601 format.

Источник