Building a Policy Q&A System: Architecture, RAG, and Practical Lessons

Description and use

A RAG‑based chatbot delivers plain‑language policy answers with source citations, improving trust and reducing support calls.

Policy documents are publicly available but unreadable for citizens, leading to support calls and guesswork.

The system delivers plain‑language policy answers with source citations and a confidence threshold, improving trust and reducing support calls.

User asks a question via React UI
Express API forwards query to LangChain retrieval
LangChain searches PostgreSQL + pgvector for relevant policy chunks
LLM generates answer grounded in retrieved chunks
System attaches source citations and confidence score
If confidence below threshold, system replies "I couldn't find a reliable answer"

Making citations a required part of every answer, not an optional extra, forced better retrieval quality from the start

The confidence threshold, even a simple version of it, cut down on confident‑sounding wrong answers a lot