Building a Production RAG Pipeline with n8n, Qdrant, and Gemini

Description and use

Building a production RAG pipeline by separating ingestion and query workflows, enriching Qdrant payloads for ACL and freshness, enforcing idempotent ingestion, and ensuring Gemini cites evidence, with robust retry and dead‑letter handling.

Problem

Production RAG systems fail when ingestion, access control, rate limits, and traceability are ignored, leading to stale answers, data leaks, and unresumable failures.

Outcome

The pipeline achieves idempotent ingestion, permission‑aware retrieval, rate‑limit‑safe model calls, traceable answers with citations, and recoverable failures, turning a demo RAG into a production‑ready system.

Workflow

• 1. Split RAG into ingestion and query pipelines in n8n.
• 2. Design Qdrant collection with payload fields: source_id, source_uri, title, chunk_index, content_hash, version, status, acl, updated_at, text.
• 3. Create payload indexes for acl, source_id, status, updated_at.
• 4. Ingest documents via POST /ingest that returns 202 and triggers async worker workflow.
• 5. Worker workflow normalizes, chunks, embeds, and upserts into Qdrant idempotently.
• 6. Query workflow receives POST /ask, embeds question, performs filtered Qdrant search, constructs prompt, calls Gemini generateContent, forces evidence citation, returns grounded answer.
• 7. Add retries, timeouts, and dead‑letter queues for both ingestion and query steps.
• 8. Implement evaluation and monitoring before exposing to users.

Evidence

Treat RAG as two separate pipelines: ingestion and query.

Source

Store more than vectors in Qdrant: source_id, acl, version, updated_at, chunk_index, and text.

Source

Make ingestion idempotent so reprocessing documents does not create duplicate truth.

Source