Your Self-Hosted AI Assistant Keeps Forgetting You. Here Is How We Measured a Fix

Описание и применение

В статье описывается, как улучшить память самодостаточного AI‑ассистента, используя SQLite для хранения, гибридный поиск (BM25 + векторный) с пороговым фильтром и ночную очистку памяти, что повышает точность и снижает фальсификации.

Задача

A self-hosted AI assistant loses or misremembers user-provided facts after a few weeks of use, leading to repeated questions, incorrect answers, and fabricated responses.

Результат

Hybrid retrieval with gating improved F1 from 0.38 to 0.48 and reduced confabulation from 0.23 to 0.03 on adversarial questions, while keeping daily model costs low.

Рабочий процесс

• 1. Store all memories, relations, and transcripts in one SQLite database in WAL mode.
• 2. Embed dates within memory facts and record rejections to preserve conflicting information.
• 3. When querying, perform BM25 keyword search and dense vector search, merge results, optionally rerank with an LLM.
• 4. Apply a relevance threshold gate; if no results exceed the gate, return no context and let the model say it does not know.
• 5. Run nightly maintenance: dedupe near-duplicates, link related memories, prune obsolete entries.
• 6. Monitor performance using the sqlite3 CLI and benchmark against LoCoMo.
• 7. Adjust thresholds and maintenance schedules based on observed F1 scores.

Доказательства

Keep the whole memory store in one file. We use SQLite: memories, relations between memories, and session transcripts all live in one database.

Источник

Hybrid retrieval fixes a surprising amount: BM25 keyword search catches the exact name, ticket number, or identifier that embeddings blur away.

Источник

The gated system scored 0.97 against 0.77 on adversarial questions with no correct answer stored.

Источник

Our estimate from the per-operation cost table is about 5 to 10 cents a day of model spend at 100 messages a day, on top of a $5 to 8 monthly VPS, or nothing extra if you run it on hardware you already have.

Источник