Your Self-Hosted AI Assistant Keeps Forgetting You. Here Is How We Measured a Fix

说明与用途

文章说明如何通过使用 SQLite 进行持久化存储、混合 BM25 与向量检索并设置相关性门限，以及夜间维护来提升自托管 AI 助手的记忆性能，从而提高准确性并减少虚构回答。

问题

A self-hosted AI assistant loses or misremembers user-provided facts after a few weeks of use, leading to repeated questions, incorrect answers, and fabricated responses.

结果

Hybrid retrieval with gating improved F1 from 0.38 to 0.48 and reduced confabulation from 0.23 to 0.03 on adversarial questions, while keeping daily model costs low.

工作流

• 1. Store all memories, relations, and transcripts in one SQLite database in WAL mode.
• 2. Embed dates within memory facts and record rejections to preserve conflicting information.
• 3. When querying, perform BM25 keyword search and dense vector search, merge results, optionally rerank with an LLM.
• 4. Apply a relevance threshold gate; if no results exceed the gate, return no context and let the model say it does not know.
• 5. Run nightly maintenance: dedupe near-duplicates, link related memories, prune obsolete entries.
• 6. Monitor performance using the sqlite3 CLI and benchmark against LoCoMo.
• 7. Adjust thresholds and maintenance schedules based on observed F1 scores.

证据

Keep the whole memory store in one file. We use SQLite: memories, relations between memories, and session transcripts all live in one database.

来源

Hybrid retrieval fixes a surprising amount: BM25 keyword search catches the exact name, ticket number, or identifier that embeddings blur away.

来源

The gated system scored 0.97 against 0.77 on adversarial questions with no correct answer stored.

来源

Our estimate from the per-operation cost table is about 5 to 10 cents a day of model spend at 100 messages a day, on top of a $5 to 8 monthly VPS, or nothing extra if you run it on hardware you already have.

来源