My embedding model couldn't find "order 48207". One-line fix and LongMemEval results

说明与用途

密集嵌入模型因失去精确标记而未能检索到特定订单号。采用混合 CombSUM 方法恢复了精确匹配，在内部测试中实现 100% 召回率，并提升了公共基准的表现。

问题

Embedding-based retrieval failed to match a specific order number in customer conversations, returning generic "late order" results.

结果

CombSUM hybrid retrieval restored exact order matching, raising recall to 100% on internal tests and improving public benchmark top‑4 recall to 52.8%. Retrieval latency dropped from 4.3 ms to 0.6 ms (p50).

工作流

• 1. Identify failure: dense embeddings miss exact token match.
• 2. Test max(model, lexical) – still fails because model scores dominate.
• 3. Implement CombSUM: sum of lexical and dense scores, ensuring neither dominates.
• 4. Add exact_match category to benchmark and unit test.
• 5. Run benchmark on 25 scenarios – 96% pass, 100% recall.
• 6. Evaluate on LongMemEval-S public dataset – top‑4 recall 52.8% overall.
• 7. Optimize retrieval: per‑customer index reduces recall latency from 4.3 ms to 0.6 ms (p50).

证据

Then another agent asks the memory: "what is going on with order 48207".

来源

To the embedding model all twelve conversations mean the same thing ("a late order"), so they scored almost identically, and recency decided which four made it into the context.

来源

With bge-m3 on all 25 scenarios: 96% pass, 100% recall, 0% leaks.

来源

Retrieval used to scan every customer's memories, then filter. A per-customer index changed the tail: recall p50 4.3 ms → 0.6 ms, recall p95 40.1 ms → 1.9 ms.

来源