How I Overhauled DocsGPT's Ingestion Engine & Supercharged RAG Context: My Hacktoberfest 2026 Journey

说明与用途

DocsGPT 解决了 24 小时源锁定问题，并通过检查点、自动恢复、孤立向量清理和实时队列监控提升了长时间导入的可靠性。新增的 CompressingRetriever 在保留代码和查询词的同时去除冗余内容。

问题

DocsGPT users were locked out of a source for 24 hours after rapid file changes, and long ingests were fragile, leading to lost work and orphaned vectors.

结果

The ingestion engine now resumes interrupted jobs, cleans orphaned vectors, and provides real‑time queue telemetry. The RAG pipeline gains a boilerplate‑pruning retriever that reduces token bloat while preserving critical content.

工作流

• Identify 24‑hour lockout issue via PR #3068
• Analyze Celery/Redis idempotency decorator and frontend SSE listener
• Replace static TTL with heartbeated lease and enforce task‑ID ownership
• Add sentinel notifications to release UI spinner
• Design ingestion reliability overhaul with checkpointing, auto‑resume, orphan sweeper, and queue gauge
• Implement CompressingRetriever with boilerplate pruning and safeguards
• Merge PRs #3252 and #3247 into main
• Validate improvements with tests and CodeRabbit AI feedback

证据

If a worker crashed, raised an uncaught network timeout to an embedding provider, or was terminated mid-task, the lock in Redis was never cleared.

来源

Database-Backed Checkpointing (0062_ingest_resume.py): Created a new IngestChunkProgress model and repository.

来源

Built an SSE-streamed queue depth and head-of-line wait time gauge so users can see exactly where their documents stand in the worker queue.

来源