The Model Reading My Benchmark Mattered More Than the Memory System Did

说明与用途

更换阅读模型后发现，评估分数差异主要来自阅读模型而非内存系统。YantrikDB 在使用 42% 更少上下文令牌的情况下实现了与 Hindsight 相当的评分，强调在评估中需报告阅读和评判模型。

问题

Benchmark results for agent-memory systems were confounded by changes in the answerer model, making it unclear whether performance gaps were due to memory architecture or reader plumbing.

结果

The reader model had a larger impact on benchmark scores than the memory architecture. YantrikDB achieved equivalent rubric scores to Hindsight using 42% fewer context tokens, but the gap was largely due to the reader. Naming reader and judge models is essential for meaningful comparisons.

工作流

• 1. Rerun BEAM‑100K benchmark with YantrikDB memory and deepseek‑v4‑flash reader. 2. Compare rubric scores to original run. 3. Replay Hindsight’s published context strings through YantrikDB’s reader and judge. 4. Perform conversation‑clustered significance testing. 5. Run a 2×2 experiment with a different reader/judge pair (kimi‑k2.6 / llama‑4‑maverick).

证据

YantrikDB’s own score moved about 9 points on that swap.

来源

The paper puts it flatly: "The published gap is therefore overwhelmingly a property of the reading model, not of the memory systems."

来源