VEKTOR 2.0.2 Memory Benchmark LongMemEval Results

说明与用途

VEKTOR 2.0.2 通过改进日期处理、增加内存回溯量和保留命名项，将 LongMemEval 准确率提升至 84%。

问题

LongMemEval benchmark showed low accuracy (84.0% on 500 questions) for VEKTOR’s memory system, especially on temporal, counting, and multi‑session questions.

结果

VEKTOR 2.0.2 achieved a 5.2 percentage point increase on the full LongMemEval benchmark, with significant gains in temporal, counting, and multi‑session question accuracy, by adjusting recall size, date handling, and named‑item saving.

工作流

• 1. Identify missed question categories via LongMemEval.
• 2. Apply Minimax matrix: test candidate settings against small sample per category, score by worst column.
• 3. Tune recall size per question type (12, 80, 120).
• 4. Add code to convert relative dates to absolute dates from conversation context.
• 5. Implement saving instruction to retain all named items.
• 6. Run paired checks with no‑change control and guard questions.
• 7. Calibrate judges and audit preference scoring.
• 8. Deploy settings via Preferences switch "Remember when things happened".

证据

With the same open models throughout, the shipped build scored 89.2% on a paired check of 120 random LongMemEval questions, against 84.2% for the build before it.

来源

Temporal questions went from 63.8% to 80.3% in the first lab run (graded by Llama 3.3 70B).

来源

Recall size now depends on the question. Plain questions return 12 memories (it was 5). Counting and totalling questions about your own history return up to 80. Order, duration and relative‑day questions return up to 120.

来源

The saving step wrote summaries and dropped the named items. One added instruction now keeps every named item from both sides of a conversation, with list items and their position, and never keeps secrets or health records.

来源

On the 52 previously missed questions it fixed 16, against 5 for a repeat with no change, and 39 of 40 right answers stayed right.

来源