I Tried to Teach AI Memory What Matters. 10 Experiments Later, Most Clever Ideas Lost

Описание и применение

Aura Memory изучила, какие части диалога сохранять, чтобы улучшить ответы. Хранение дословных сообщений пользователя оказалось наиболее эффективным, а обучение забывания на основе последствий повысило точность на 16 %.

Задача

Determining which parts of a conversation should be stored in an AI agent’s memory to maximize future answer quality while keeping memory size manageable.

Результат

Storing verbatim user messages yields 79% correct answers on LongMemEval, far better than summaries or no storage. 25% turn retention with learned forgetting based on consequence labels improves recall by 16 points over time‑based decay. Real outcome signals from humans (corrections, repeats, commits) are essential for learning what matters.

Рабочий процесс

• 1. Run LongMemEval with different storage strategies (none, user messages, all, LLM summaries, user+summaries).
• 2. Label 319 real user messages for durability (facts, preferences, decisions, corrections).
• 3. Stream long conversations (LoCoMo) with a 25% turn budget and test forgetting rules (no forgetting, decay, oldest, least retrieved).
• 4. Train a small network on consequence labels from 5 conversations to predict useful records, then test on 5 other conversations.
• 5. Evaluate signal usefulness on 67,000 OpenHands trajectories (exit codes, test passes, combined signals).
• 6. Apply lessons: keep newer facts, use a control facts checker, and deliver preferences directly to the model.
• 7. Release Aura Memory 1.60.1 with data‑loss fix and plan 1.61 features (recall separation, security profiles, source resistance, MCP server, privacy scopes).

Доказательства

the user's words carry almost all the value at an eighth of the size

Источник

45% carry something durable: facts about my setup (64), preferences (34), decisions with reasons (30), corrections (14).

Источник

+16 points over time-based decay; +13.5 points on average across 6 different train/test splits

Источник

exit codes of its commands 0.51–0.53; all signals together 0.66

Источник