How I Overhauled DocsGPT's Ingestion Engine & Supercharged RAG Context: My Hacktoberfest 2026 Journey

Description and use

DocsGPT fixed a 24‑hour source lockout and improved long‑run ingestion reliability with checkpoints, auto‑resume, orphan cleanup, and real‑time queue telemetry. A new CompressingRetriever prunes boilerplate while preserving code and query terms.

Problem

DocsGPT users were locked out of a source for 24 hours after rapid file changes, and long ingests were fragile, leading to lost work and orphaned vectors.

Outcome

The ingestion engine now resumes interrupted jobs, cleans orphaned vectors, and provides real‑time queue telemetry. The RAG pipeline gains a boilerplate‑pruning retriever that reduces token bloat while preserving critical content.

Workflow

• Identify 24‑hour lockout issue via PR #3068
• Analyze Celery/Redis idempotency decorator and frontend SSE listener
• Replace static TTL with heartbeated lease and enforce task‑ID ownership
• Add sentinel notifications to release UI spinner
• Design ingestion reliability overhaul with checkpointing, auto‑resume, orphan sweeper, and queue gauge
• Implement CompressingRetriever with boilerplate pruning and safeguards
• Merge PRs #3252 and #3247 into main
• Validate improvements with tests and CodeRabbit AI feedback

Evidence

If a worker crashed, raised an uncaught network timeout to an embedding provider, or was terminated mid-task, the lock in Redis was never cleared.

Source

Database-Backed Checkpointing (0062_ingest_resume.py): Created a new IngestChunkProgress model and repository.

Source

Built an SSE-streamed queue depth and head-of-line wait time gauge so users can see exactly where their documents stand in the worker queue.

Source