The Model Reading My Benchmark Mattered More Than the Memory System Did

Описание и применение

В результате замены модели чтения контекста в оценке памяти агента обнаружено, что большая часть разницы в баллах связана с моделью чтения, а не с архитектурой памяти. При использовании более компактного контекста YantrikDB достигает эквивалентных результатов, но с 42 % меньшим количеством токенов.

Задача

Benchmark results for agent-memory systems were confounded by changes in the answerer model, making it unclear whether performance gaps were due to memory architecture or reader plumbing.

Результат

The reader model had a larger impact on benchmark scores than the memory architecture. YantrikDB achieved equivalent rubric scores to Hindsight using 42% fewer context tokens, but the gap was largely due to the reader. Naming reader and judge models is essential for meaningful comparisons.

Рабочий процесс

• 1. Rerun BEAM‑100K benchmark with YantrikDB memory and deepseek‑v4‑flash reader. 2. Compare rubric scores to original run. 3. Replay Hindsight’s published context strings through YantrikDB’s reader and judge. 4. Perform conversation‑clustered significance testing. 5. Run a 2×2 experiment with a different reader/judge pair (kimi‑k2.6 / llama‑4‑maverick).

Доказательства

YantrikDB’s own score moved about 9 points on that swap.

Источник

The paper puts it flatly: "The published gap is therefore overwhelmingly a property of the reading model, not of the memory systems."

Источник