A docs assistant a small business can run on its own laptop, built with open-weight AI

Description and use

Small businesses can deploy a local docs assistant that retrieves and cites answers from their own manuals, preserving privacy and avoiding per‑token costs.

Problem

Small businesses with technical products repeatedly answer the same support questions from internal manuals, but their staff manually search PDFs and wikis and cannot easily use closed AI services.

Outcome

The assistant enables small businesses to automate support queries locally, preserving privacy, eliminating per‑token costs, and avoiding vendor lock‑in.

Workflow

• 1. Install the assistant on the customer’s machine or server.
• 2. Load the customer’s documents into the system.
• 3. The assistant indexes document chunks using the BAAI/bge-small-en-v1.5 embedding model via fastembed and stores vectors in PostgreSQL with pgvector.
• 4. When a user asks a question, the assistant performs a hybrid search (vector + keyword) to find relevant pages.
• 5. The open‑weight LLM (Qwen2.5 1.5B/3B 4‑bit served by mlx‑lm) generates an answer using only the retrieved pages, adding citations.
• 6. The system logs each request and displays answered/unanswered status on a dashboard.
• 7. The customer can update documents and rerun tests to measure performance.
• 8. The customer pays a one‑time setup fee and a small monthly fee for document updates.

Evidence

PumpGuard assistant: a question-and-answer assistant that answers from a company's own documents (manuals, FAQs, error-code tables), cites the page it used, and says "I could not find that in the knowledge base." when the answer isn't there.

Source

Each chunk of the knowledge base is turned into a vector with BAAI/bge-small-en-v1.5 through fastembed, and stored in PostgreSQL with pgvector.

Source

I ran it with Qwen2.5 (0.5B, 1.5B and 3B, 4-bit) served by mlx-lm on my 16 GB Mac, and compared the sizes on the same questions.

Source

The writer may answer only from the retrieved pages, every answer carries citations, a role filter keeps staff-only pages (error codes) away from customers, and an audit log records each request.

Source