Manage LinkedIn Pipeline State with Python and n8n

说明与用途

使用 Apify 和 n8n 的自动化工作流高效收集 LinkedIn 数据，避免重复、超时和缺失字段错误。

问题

Automating LinkedIn profile scraping at scale leads to redundant processing, timeouts, and schema inconsistencies when using Apify’s linkedin-profile-scraper.

结果

The pipeline now processes only new or expired profiles, avoids 5‑minute timeouts by using asynchronous runs, receives results via webhooks, and handles missing data without crashes.

工作流

• 1. Store processed profile URLs and last scraped timestamps in SQLite/PostgreSQL.
• 2. Before each run, query the database to compute set difference and produce a list of new/expired URLs.
• 3. Trigger the Apify actor asynchronously with a POST request to the runs endpoint, capturing the run ID.
• 4. Configure an Apify webhook for ACTOR.RUN.SUCCEEDED to POST to an n8n HTTP trigger node.
• 5. In n8n, parse the webhook payload, extract the dataset ID, and download the dataset via Apify API.
• 6. Process each profile JSON with Python, using .get() to handle missing keys and log warnings for redacted fields.
• 7. Store cleaned data into downstream systems.
• 8. Update the local database with the new last‑scraped timestamps.

证据

To avoid double processing LinkedIn profiles, you must implement an external state-tracking layer.

来源

Apify enforces a strict 300-second hard cap on synchronous run API calls, returning an HTTP 408 response code.

来源

The scraper includes safeguards like discarding asterisk-masked text, but your processing logic must expect and handle missing schema items as a normal operational state.

来源