Solving Context Rot: Integrating Tencent/WeKnora for Cache-Aligned RAG and Token Control

Description and use

Unstable context in RAG pipelines caused a 400% billing spike and 9.4 s latency. Integrating Tencent/WeKnora and deterministic prefix canonicalization restored cache hits to ~80%, reduced token usage, and returned latency to ~1.1 s.

Problem

Naive RAG agents concatenated 32,000 un‑deduplicated tokens per query, causing a 400% spike in LLM billing, 9.4 s latency, and zero prompt‑cache hits.

Outcome

Deterministic prefix alignment restored prompt cache hit rates to ~80%+, reduced token consumption, and stabilized latency from 9.4 s to ~1.1 s while preventing a 400% billing spike.

Workflow

• 1. Ingest raw docs/RFCs into Tencent/WeKnora to extract entities and build wiki graph.
• 2. Canonicalize entities alphabetically and assemble a static prefix with system manifest and entity block.
• 3. Deduplicate retrieved chunks using SHA‑256 hashes, select up to 5 unique chunks, and assemble a dynamic suffix.
• 4. Combine static prefix and dynamic suffix to form the final prompt payload.
• 5. Route payload through cache‑aligned gateway to upstream LLM.
• 6. Batch wiki syncs hourly to avoid immediate cache invalidation.
• 7. Monitor latency, token usage, and cache hit rates.
• 8. Adjust chunk budget and reranker precision to balance latency and reasoning depth.

Evidence

By segregating the canonical knowledge base definition from dynamic query evidence, the first 12,000 tokens of our system prompt stay invariant across millions of daily queries, allowing upstream gateways to service them out of KV cache.

Source

We had to enforce an hourly batching window for wiki syncs rather than streaming instant updates.

Source