The Model Reading My Benchmark Mattered More Than the Memory System Did

Description and use

Swapping the reader model in the agent‑memory benchmark revealed that the reader, not the memory system, drives most score differences. YantrikDB matches Hindsight’s rubric score using 42% fewer context tokens, underscoring the need to report reader and judge models in evaluations.

Problem

Benchmark results for agent-memory systems were confounded by changes in the answerer model, making it unclear whether performance gaps were due to memory architecture or reader plumbing.

Outcome

The reader model had a larger impact on benchmark scores than the memory architecture. YantrikDB achieved equivalent rubric scores to Hindsight using 42% fewer context tokens, but the gap was largely due to the reader. Naming reader and judge models is essential for meaningful comparisons.

Workflow

• 1. Rerun BEAM‑100K benchmark with YantrikDB memory and deepseek‑v4‑flash reader. 2. Compare rubric scores to original run. 3. Replay Hindsight’s published context strings through YantrikDB’s reader and judge. 4. Perform conversation‑clustered significance testing. 5. Run a 2×2 experiment with a different reader/judge pair (kimi‑k2.6 / llama‑4‑maverick).

Evidence

YantrikDB’s own score moved about 9 points on that swap.

Source

The paper puts it flatly: "The published gap is therefore overwhelmingly a property of the reading model, not of the memory systems."

Source