Build, Adopt, Build Again: Seven Months of Search in a Personal AI Tool

Description and use

Migrating Open Walnut’s search to a custom tokenizer and BM25 engine reduced latency to <200 ms and improved recall for mixed-language and code-heavy data.

Problem

The original search engine in Open Walnut was inadequate for mixed-language, code-heavy data, causing poor recall, high latency, and extensive maintenance.

Outcome

The migration to a custom tokenizer and BM25 engine reduced average search latency from ~4 s to <200 ms, eliminated the need for external vector services, and improved recall for mixed-language and code‑identifier queries.

Workflow

• 1. Built a golden‑set harness with real queries and assertions.
• 2. Evaluated the existing QMD engine to establish a baseline.
• 3. Designed a custom tokenizer emitting original and sub‑token streams.
• 4. Implemented the tokenizer at index and query time using SQLite FTS5.
• 5. Replaced the library with the homegrown engine and removed external processes.
• 6. Benchmarked recall, latency, and resource usage.
• 7. Deployed the new engine with a feature flag and rolled out to production.

Evidence

The operational cost grew with it: /api/search averaged 4,034 ms in production. The 300 ms skill prefetch almost never won.

Source

We moved tokenization into our own code, applied the same way at index time and query time.

Source

The tokenizer is a single pass over character codes that emits two ordered streams.

Source