I Built a Self-Hosted RAG Stack That Runs Without Cloud AI APIs

Description and use

LocalRAG is a fully local RAG stack that uploads documents, generates embeddings, and answers questions without cloud API calls, enabling use in confidential and isolated environments.

Problem

Teams need document question‑answering without sending data to cloud AI services.

Outcome

LocalRAG successfully indexed a fictional employee handbook into three chunks and answered a leave‑policy question with 15 days, displaying source passages and similarity scores.

Workflow

• Upload PDF/DOCX/TXT/Markdown
• Parse and split into chunks
• Generate embeddings with local bge‑m3 via Ollama
• Store vectors in Qdrant
• Embed user question
• Retrieve relevant chunks
• Generate answer with local LLM (qwen2.5:3b) via Ollama
• Return answer with source references

Evidence

Many teams want AI to answer questions from internal documents, but they cannot send those documents to a cloud service.

Source

LocalRAG allows a user to upload documents and ask questions against them.

Source

The application does not call a cloud AI API at runtime.

Source

I tested the current release with: Docker Desktop 4.95.0, Ollama 0.11.3, Qdrant, bge-m3 for embeddings, qwen2.5:3b for generation.

Source

The system: Indexed it into three chunks. Received the question: “How many days of annual leave do full‑time employees receive?” Retrieved the relevant policy text. Returned “15 days.”

Source