I built a local AI memory gate on a CPU, and my 7B model scored worse than my 1.5B model because it was too smart

Description and use

A lightweight offline memory layer for local LLMs using SQLite, Hebbian plasticity, and hyperdimensional vectors; 1.5B Qwen scored 50% accuracy, while 5.2B Qwen scored 25% due to strict evaluation.

Problem

Creating a lightweight, offline memory layer for local LLMs that avoids heavy vector databases and VRAM waste.

Outcome

The 1.5B Qwen model achieved 50% retrieval accuracy, while the 5.2B Qwen model, despite richer outputs, scored only 25% due to rigid evaluation expectations.

Workflow

• 1. Store facts as SQLite triples (Subject-Predicate-Object).
• 2. Track co-occurring concepts via Hebbian plasticity to build associative weights.
• 3. Maintain a 10,000‑dimensional leaky context vector for pronoun resolution and query gating.
• 4. Ingest conversation, extract triples, and update the database.
• 5. Run retrieval accuracy benchmark on Qwen models.

Evidence

When I ran Qwen 2 (1.5B), it got around 50.0% Retrieval Accuracy. But when I upgraded to the much smarter Qwen 3 (5.2GB), the ingestion time jumped to 20 minutes on my local machine, and its score actually dropped to 25.0%!

Source

The codebase is written in pure Python, is fully open-source (under the AGPL-3.0 copyleft license), and is designed to run entirely offline on consumer hardware.

Source