Local RAG: querying my documents without the cloud (2026)

Description and use

The author built a local RAG pipeline to query personal documents without uploading them to the cloud, preserving privacy and model control.

Problem

The author needed to query a large collection of personal documents (runbooks, notes, tickets) without uploading them to a cloud service, to avoid privacy concerns and control over data and model updates.

Outcome

The author achieved a fully local RAG pipeline that allows querying personal documents without cloud exposure, reducing costs, ensuring data privacy, and maintaining control over model updates.

Workflow

• 1. Set up a Docker environment with a local LLM runtime and web UI.
• 2. Configure the Docker compose file to point the UI to a folder containing documents.
• 3. Upload documents (PDF, Markdown, text, office files) via the UI; the system splits, embeds, and stores them in a vector store.
• 4. Query the system through the chat interface; the LLM retrieves relevant chunks from the vector store and generates grounded answers.
• 5. All data remains encrypted on local disk; no cloud interaction occurs.

Evidence

I built a local RAG pipeline in my home lab and now I query my own searchable knowledge base across my LAN, and the files never leave my house.

Source

If you already run quantized models through a web UI and Docker is second nature, the extra step is just pointing the UI at a folder of documents and letting it build the index.

Source