Building a Production RAG Pipeline with n8n, Qdrant, and Gemini

说明与用途

通过将 RAG 分为摄取和查询两条流水线，使用 n8n 编排、Qdrant 存储带权限和新鲜度的元数据，并强制 Gemini 引用证据，构建可生产的 RAG 流程，并加入重试和死信队列。

问题

Production RAG systems fail when ingestion, access control, rate limits, and traceability are ignored, leading to stale answers, data leaks, and unresumable failures.

结果

The pipeline achieves idempotent ingestion, permission‑aware retrieval, rate‑limit‑safe model calls, traceable answers with citations, and recoverable failures, turning a demo RAG into a production‑ready system.

工作流

• 1. Split RAG into ingestion and query pipelines in n8n.
• 2. Design Qdrant collection with payload fields: source_id, source_uri, title, chunk_index, content_hash, version, status, acl, updated_at, text.
• 3. Create payload indexes for acl, source_id, status, updated_at.
• 4. Ingest documents via POST /ingest that returns 202 and triggers async worker workflow.
• 5. Worker workflow normalizes, chunks, embeds, and upserts into Qdrant idempotently.
• 6. Query workflow receives POST /ask, embeds question, performs filtered Qdrant search, constructs prompt, calls Gemini generateContent, forces evidence citation, returns grounded answer.
• 7. Add retries, timeouts, and dead‑letter queues for both ingestion and query steps.
• 8. Implement evaluation and monitoring before exposing to users.

证据

Treat RAG as two separate pipelines: ingestion and query.

来源

Store more than vectors in Qdrant: source_id, acl, version, updated_at, chunk_index, and text.

来源

Make ingestion idempotent so reprocessing documents does not create duplicate truth.

来源