Building Document Intelligence: A Document Q&A API with FastAPI, Qdrant, and RAG

Описание и применение

Система автоматически загружает, извлекает, индексирует и отвечает на вопросы по финансовым документам разных форматов, используя FastAPI, Celery, MongoDB, Qdrant и модели OpenAI/Gemini.

Задача

Financial information is spread across PDFs, spreadsheets, scanned receipts, and reports, making manual search tedious.

Результат

The system enables automated ingestion, extraction, indexing, and question answering over heterogeneous financial documents, reducing manual effort and providing source-backed answers.

Рабочий процесс

• Upload documents via POST /files
• Validate files and create MongoDB record
• Store original in S3-compatible storage
• Queue Celery task
• Celery worker retrieves file and selects parser by extension
• Extract text using format-specific parsers (PyMuPDF, OCRmyPDF, Tesseract, etc.)
• Classify PDFs into native-text, image-based, or hybrid and apply appropriate OCR
• Chunk extracted content by paragraph boundaries
• Generate dense embeddings with BAAI/bge-m3 and sparse embeddings with Qdrant/bm25
• Store chunks and vectors in MongoDB and Qdrant
• When a question arrives, embed query in both spaces
• Retrieve candidate chunks, fuse with reciprocal-rank, optionally rerank with BAAI/bge-reranker-v2-m3
• Return top chunks
• Generate answer with OpenAI/Gemini via LangChain, using retrieved chunks as context
• Return answer and source chunks (optionally stream via SSE)

Доказательства

Financial information is often distributed across different document formats.

Источник

The application prompt instructs the model to use the supplied context and avoid presenting financial advice as fact.

Источник

The main components are: FastAPI, Celery, Redis, MongoDB, MinIO AIStor / S3, Qdrant, Sentence Transformers, FastEmbed, OpenAI / Gemini, PyMuPDF / OCRmyPDF / Tesseract.

Источник