Building an automated web ingestion pipeline in n8n for LLM knowledge bases

说明与用途

使用 n8n 的自动化管道，利用 MESSORA 提取网页的干净 Markdown，生成 OpenAI 嵌入，并更新 Qdrant/Pinecone 向量存储，绕过 WAF 并减少令牌使用量。

问题

Automating weekly competitor intelligence, price tracking, or documentation updates in n8n often fails when scrapers hit anti‑bot challenges or return uncleaned HTML that exceeds context boundaries.

结果

The pipeline runs daily, bypasses anti‑bot challenges, outputs clean Markdown, reduces token usage by ~75%, and automatically updates the vector store without managing headless browsers.

工作流

• Schedule Trigger (Daily at 08:00)
• Read URL List from Google Sheet
• HTTP Request: POST /v1/extract (MESSORA)
• Generate Embeddings (OpenAI)
• Upsert to Vector Store (Qdrant / Pinecone)

证据

Zero infrastructure management: No headless Chromium instances hanging on your n8n host.

来源

75% token reduction: Ingests clean Markdown instead of full DOM hierarchies.

来源