VEKTOR 2.0.2 Memory Benchmark LongMemEval Results

Description and use

VEKTOR 2.0.2 raised LongMemEval accuracy to 84 % by improving date handling, increasing memory recall, and preserving named items.

Problem

LongMemEval benchmark showed low accuracy (84.0% on 500 questions) for VEKTOR’s memory system, especially on temporal, counting, and multi‑session questions.

Outcome

VEKTOR 2.0.2 achieved a 5.2 percentage point increase on the full LongMemEval benchmark, with significant gains in temporal, counting, and multi‑session question accuracy, by adjusting recall size, date handling, and named‑item saving.

Workflow

• 1. Identify missed question categories via LongMemEval.
• 2. Apply Minimax matrix: test candidate settings against small sample per category, score by worst column.
• 3. Tune recall size per question type (12, 80, 120).
• 4. Add code to convert relative dates to absolute dates from conversation context.
• 5. Implement saving instruction to retain all named items.
• 6. Run paired checks with no‑change control and guard questions.
• 7. Calibrate judges and audit preference scoring.
• 8. Deploy settings via Preferences switch "Remember when things happened".

Evidence

With the same open models throughout, the shipped build scored 89.2% on a paired check of 120 random LongMemEval questions, against 84.2% for the build before it.

Source

Temporal questions went from 63.8% to 80.3% in the first lab run (graded by Llama 3.3 70B).

Source

Recall size now depends on the question. Plain questions return 12 memories (it was 5). Counting and totalling questions about your own history return up to 80. Order, duration and relative‑day questions return up to 120.

Source

The saving step wrote summaries and dropped the named items. One added instruction now keeps every named item from both sides of a conversation, with list items and their position, and never keeps secrets or health records.

Source

On the 52 previously missed questions it fixed 16, against 5 for a repeat with no change, and 39 of 40 right answers stayed right.

Source