Building an automated web ingestion pipeline in n8n for LLM knowledge bases

Description and use

An automated n8n pipeline uses MESSORA to extract clean Markdown from web pages, generates OpenAI embeddings, and updates a vector store (Qdrant/Pinecone), bypassing WAFs and cutting token usage.

Problem

Automating weekly competitor intelligence, price tracking, or documentation updates in n8n often fails when scrapers hit anti‑bot challenges or return uncleaned HTML that exceeds context boundaries.

Outcome

The pipeline runs daily, bypasses anti‑bot challenges, outputs clean Markdown, reduces token usage by ~75%, and automatically updates the vector store without managing headless browsers.

Workflow

• Schedule Trigger (Daily at 08:00)
• Read URL List from Google Sheet
• HTTP Request: POST /v1/extract (MESSORA)
• Generate Embeddings (OpenAI)
• Upsert to Vector Store (Qdrant / Pinecone)

Evidence

Zero infrastructure management: No headless Chromium instances hanging on your n8n host.

Source

75% token reduction: Ingests clean Markdown instead of full DOM hierarchies.

Source