I Tried to Teach AI Memory What Matters. 10 Experiments Later, Most Clever Ideas Lost

Description and use

Aura Memory discovered that storing verbatim user messages and learning forgetting policies from real consequences improves recall by 16 % over time‑based decay.

Problem

Determining which parts of a conversation should be stored in an AI agent’s memory to maximize future answer quality while keeping memory size manageable.

Outcome

Storing verbatim user messages yields 79% correct answers on LongMemEval, far better than summaries or no storage. 25% turn retention with learned forgetting based on consequence labels improves recall by 16 points over time‑based decay. Real outcome signals from humans (corrections, repeats, commits) are essential for learning what matters.

Workflow

• 1. Run LongMemEval with different storage strategies (none, user messages, all, LLM summaries, user+summaries).
• 2. Label 319 real user messages for durability (facts, preferences, decisions, corrections).
• 3. Stream long conversations (LoCoMo) with a 25% turn budget and test forgetting rules (no forgetting, decay, oldest, least retrieved).
• 4. Train a small network on consequence labels from 5 conversations to predict useful records, then test on 5 other conversations.
• 5. Evaluate signal usefulness on 67,000 OpenHands trajectories (exit codes, test passes, combined signals).
• 6. Apply lessons: keep newer facts, use a control facts checker, and deliver preferences directly to the model.
• 7. Release Aura Memory 1.60.1 with data‑loss fix and plan 1.61 features (recall separation, security profiles, source resistance, MCP server, privacy scopes).

Evidence

the user's words carry almost all the value at an eighth of the size

Source

45% carry something durable: facts about my setup (64), preferences (34), decisions with reasons (30), corrections (14).

Source

+16 points over time-based decay; +13.5 points on average across 6 different train/test splits

Source

exit codes of its commands 0.51–0.53; all signals together 0.66

Source