How I Overhauled DocsGPT's Ingestion Engine & Supercharged RAG Context: My Hacktoberfest 2026 Journey

Описание и применение

В DocsGPT была устранена проблема блокировки источника на 24 часа и повышена надёжность долгих загрузок, добавлены контрольные точки, автоматическое возобновление и очистка орфанных векторов. Также внедрён компрессор RAG, удаляющий лишний контент, сохраняя при этом код и запросы.

Задача

DocsGPT users were locked out of a source for 24 hours after rapid file changes, and long ingests were fragile, leading to lost work and orphaned vectors.

Результат

The ingestion engine now resumes interrupted jobs, cleans orphaned vectors, and provides real‑time queue telemetry. The RAG pipeline gains a boilerplate‑pruning retriever that reduces token bloat while preserving critical content.

Рабочий процесс

• Identify 24‑hour lockout issue via PR #3068
• Analyze Celery/Redis idempotency decorator and frontend SSE listener
• Replace static TTL with heartbeated lease and enforce task‑ID ownership
• Add sentinel notifications to release UI spinner
• Design ingestion reliability overhaul with checkpointing, auto‑resume, orphan sweeper, and queue gauge
• Implement CompressingRetriever with boilerplate pruning and safeguards
• Merge PRs #3252 and #3247 into main
• Validate improvements with tests and CodeRabbit AI feedback

Доказательства

If a worker crashed, raised an uncaught network timeout to an embedding provider, or was terminated mid-task, the lock in Redis was never cleared.

Источник

Database-Backed Checkpointing (0062_ingest_resume.py): Created a new IngestChunkProgress model and repository.

Источник

Built an SSE-streamed queue depth and head-of-line wait time gauge so users can see exactly where their documents stand in the worker queue.

Источник