How I Built an Autonomous B2B Directory Scraping & Enrichment Pipeline with n8n and Python

Description and use

An autonomous B2B directory scraping and enrichment pipeline built with n8n, Playwright, and Pydantic that addresses dynamic rendering, orchestration coupling, and schema drift.

Problem

Traditional B2B directory scrapers fail due to client‑side hydration, anti‑automation fingerprints, orchestration coupling, and schema drift.

Outcome

The pipeline achieves autonomous, real‑time B2B directory enrichment with robust error handling, schema enforcement, and multi‑destination support.

Workflow

• 1. n8n pulls a page URL from a queue.
• 2. n8n sends the URL to a FastAPI /extract endpoint.
• 3. Playwright launches a stealth browser, scrolls to trigger hydration, and extracts company cards.
• 4. Extracted raw data is validated and normalized by Pydantic.
• 5. n8n deduplicates and writes verified records to PostgreSQL, Airtable, or a CRM.

Evidence

Modern directory listings defer rendering contact cards and metadata until specific micro‑interactions occur (scroll events, viewport visibility).

Source

Embedding proxy rotation, retry backoff, browser pools, and CRM export logic inside a single monolithic Python script creates unmaintainable code.

Source

Pydantic Validation Layer – RFC Email validation, E.164 Phone formatting, Domain sanitization.

Source