Solving Context Rot: Integrating Tencent/WeKnora for Cache-Aligned RAG and Token Control

说明与用途

RAG管道中的不稳定上下文导致费用激增和延迟升高。通过集成Tencent/WeKnora并实施确定性前缀规范化，恢复了约80%的缓存命中率，降低了令牌使用量，并将延迟恢复到约1.1秒。

问题

Naive RAG agents concatenated 32,000 un‑deduplicated tokens per query, causing a 400% spike in LLM billing, 9.4 s latency, and zero prompt‑cache hits.

结果

Deterministic prefix alignment restored prompt cache hit rates to ~80%+, reduced token consumption, and stabilized latency from 9.4 s to ~1.1 s while preventing a 400% billing spike.

工作流

• 1. Ingest raw docs/RFCs into Tencent/WeKnora to extract entities and build wiki graph.
• 2. Canonicalize entities alphabetically and assemble a static prefix with system manifest and entity block.
• 3. Deduplicate retrieved chunks using SHA‑256 hashes, select up to 5 unique chunks, and assemble a dynamic suffix.
• 4. Combine static prefix and dynamic suffix to form the final prompt payload.
• 5. Route payload through cache‑aligned gateway to upstream LLM.
• 6. Batch wiki syncs hourly to avoid immediate cache invalidation.
• 7. Monitor latency, token usage, and cache hit rates.
• 8. Adjust chunk budget and reranker precision to balance latency and reasoning depth.

证据

By segregating the canonical knowledge base definition from dynamic query evidence, the first 12,000 tokens of our system prompt stay invariant across millions of daily queries, allowing upstream gateways to service them out of KV cache.

来源

We had to enforce an hourly batching window for wiki syncs rather than streaming instant updates.

来源