Building a Production RAG Pipeline with n8n, Qdrant, and Gemini

Описание и применение

Построение производственной RAG‑конвейера с разделением на два пайплайна: ingestion и query, использование n8n для оркестрации, Qdrant с расширенными метаданными для контроля доступа и Gemini для генерации ответов с цитатами.

Задача

Production RAG systems fail when ingestion, access control, rate limits, and traceability are ignored, leading to stale answers, data leaks, and unresumable failures.

Результат

The pipeline achieves idempotent ingestion, permission‑aware retrieval, rate‑limit‑safe model calls, traceable answers with citations, and recoverable failures, turning a demo RAG into a production‑ready system.

Рабочий процесс

• 1. Split RAG into ingestion and query pipelines in n8n.
• 2. Design Qdrant collection with payload fields: source_id, source_uri, title, chunk_index, content_hash, version, status, acl, updated_at, text.
• 3. Create payload indexes for acl, source_id, status, updated_at.
• 4. Ingest documents via POST /ingest that returns 202 and triggers async worker workflow.
• 5. Worker workflow normalizes, chunks, embeds, and upserts into Qdrant idempotently.
• 6. Query workflow receives POST /ask, embeds question, performs filtered Qdrant search, constructs prompt, calls Gemini generateContent, forces evidence citation, returns grounded answer.
• 7. Add retries, timeouts, and dead‑letter queues for both ingestion and query steps.
• 8. Implement evaluation and monitoring before exposing to users.

Доказательства

Treat RAG as two separate pipelines: ingestion and query.

Источник

Store more than vectors in Qdrant: source_id, acl, version, updated_at, chunk_index, and text.

Источник

Make ingestion idempotent so reprocessing documents does not create duplicate truth.

Источник