How I Built an Autonomous B2B Directory Scraping & Enrichment Pipeline with n8n and Python

Описание и применение

Автономный пайплайн для сбора и обогащения B2B‑директорий, использующий n8n, Playwright и Pydantic, решает проблемы динамического рендеринга, оркестрации и качества данных.

Задача

Traditional B2B directory scrapers fail due to client‑side hydration, anti‑automation fingerprints, orchestration coupling, and schema drift.

Результат

The pipeline achieves autonomous, real‑time B2B directory enrichment with robust error handling, schema enforcement, and multi‑destination support.

Рабочий процесс

• 1. n8n pulls a page URL from a queue.
• 2. n8n sends the URL to a FastAPI /extract endpoint.
• 3. Playwright launches a stealth browser, scrolls to trigger hydration, and extracts company cards.
• 4. Extracted raw data is validated and normalized by Pydantic.
• 5. n8n deduplicates and writes verified records to PostgreSQL, Airtable, or a CRM.

Доказательства

Modern directory listings defer rendering contact cards and metadata until specific micro‑interactions occur (scroll events, viewport visibility).

Источник

Embedding proxy rotation, retry backoff, browser pools, and CRM export logic inside a single monolithic Python script creates unmaintainable code.

Источник

Pydantic Validation Layer – RFC Email validation, E.164 Phone formatting, Domain sanitization.

Источник