Manage LinkedIn Pipeline State with Python and n8n

Описание и применение

Поток автоматизации, использующий Apify и n8n, позволяет эффективно собирать данные LinkedIn, избегая дублирования, тайм‑аутов и ошибок из‑за отсутствующих полей.

Задача

Automating LinkedIn profile scraping at scale leads to redundant processing, timeouts, and schema inconsistencies when using Apify’s linkedin-profile-scraper.

Результат

The pipeline now processes only new or expired profiles, avoids 5‑minute timeouts by using asynchronous runs, receives results via webhooks, and handles missing data without crashes.

Рабочий процесс

• 1. Store processed profile URLs and last scraped timestamps in SQLite/PostgreSQL.
• 2. Before each run, query the database to compute set difference and produce a list of new/expired URLs.
• 3. Trigger the Apify actor asynchronously with a POST request to the runs endpoint, capturing the run ID.
• 4. Configure an Apify webhook for ACTOR.RUN.SUCCEEDED to POST to an n8n HTTP trigger node.
• 5. In n8n, parse the webhook payload, extract the dataset ID, and download the dataset via Apify API.
• 6. Process each profile JSON with Python, using .get() to handle missing keys and log warnings for redacted fields.
• 7. Store cleaned data into downstream systems.
• 8. Update the local database with the new last‑scraped timestamps.

Доказательства

To avoid double processing LinkedIn profiles, you must implement an external state-tracking layer.

Источник

Apify enforces a strict 300-second hard cap on synchronous run API calls, returning an HTTP 408 response code.

Источник

The scraper includes safeguards like discarding asterisk-masked text, but your processing logic must expect and handle missing schema items as a normal operational state.

Источник