Building a Policy Q&A System: Architecture, RAG, and Practical Lessons

Описание и применение

Система чат‑бота на основе RAG предоставляет гражданам простые ответы на вопросы о публичных политиках с точными ссылками на источники, повышая доверие и снижая нагрузку на службы поддержки.

Задача

Policy documents are publicly available but unreadable for citizens, leading to support calls and guesswork.

Результат

The system delivers plain‑language policy answers with source citations and a confidence threshold, improving trust and reducing support calls.

Рабочий процесс

• User asks a question via React UI
• Express API forwards query to LangChain retrieval
• LangChain searches PostgreSQL + pgvector for relevant policy chunks
• LLM generates answer grounded in retrieved chunks
• System attaches source citations and confidence score
• If confidence below threshold, system replies "I couldn't find a reliable answer"

Доказательства

Making citations a required part of every answer, not an optional extra, forced better retrieval quality from the start

Источник

The confidence threshold, even a simple version of it, cut down on confident‑sounding wrong answers a lot

Источник