My embedding model couldn't find "order 48207". One-line fix and LongMemEval results

Description and use

Dense embeddings missed a specific order number due to token loss. A hybrid CombSUM approach restored exact matching, achieving 100 % recall on internal tests and improving public benchmark results.

Problem

Embedding-based retrieval failed to match a specific order number in customer conversations, returning generic "late order" results.

Outcome

CombSUM hybrid retrieval restored exact order matching, raising recall to 100% on internal tests and improving public benchmark top‑4 recall to 52.8%. Retrieval latency dropped from 4.3 ms to 0.6 ms (p50).

Workflow

• 1. Identify failure: dense embeddings miss exact token match.
• 2. Test max(model, lexical) – still fails because model scores dominate.
• 3. Implement CombSUM: sum of lexical and dense scores, ensuring neither dominates.
• 4. Add exact_match category to benchmark and unit test.
• 5. Run benchmark on 25 scenarios – 96% pass, 100% recall.
• 6. Evaluate on LongMemEval-S public dataset – top‑4 recall 52.8% overall.
• 7. Optimize retrieval: per‑customer index reduces recall latency from 4.3 ms to 0.6 ms (p50).

Evidence

Then another agent asks the memory: "what is going on with order 48207".

Source

To the embedding model all twelve conversations mean the same thing ("a late order"), so they scored almost identically, and recency decided which four made it into the context.

Source

With bge-m3 on all 25 scenarios: 96% pass, 100% recall, 0% leaks.

Source

Retrieval used to scan every customer's memories, then filter. A per-customer index changed the tail: recall p50 4.3 ms → 0.6 ms, recall p95 40.1 ms → 1.9 ms.

Source