Build, Adopt, Build Again: Seven Months of Search in a Personal AI Tool

Описание и применение

Перенос поисковой системы в Open Walnut с использованием собственного токенизатора и BM25 снизил задержку до <200 мс и повысил точность поиска в многоязычной и кодовой среде.

Задача

The original search engine in Open Walnut was inadequate for mixed-language, code-heavy data, causing poor recall, high latency, and extensive maintenance.

Результат

The migration to a custom tokenizer and BM25 engine reduced average search latency from ~4 s to <200 ms, eliminated the need for external vector services, and improved recall for mixed-language and code‑identifier queries.

Рабочий процесс

• 1. Built a golden‑set harness with real queries and assertions.
• 2. Evaluated the existing QMD engine to establish a baseline.
• 3. Designed a custom tokenizer emitting original and sub‑token streams.
• 4. Implemented the tokenizer at index and query time using SQLite FTS5.
• 5. Replaced the library with the homegrown engine and removed external processes.
• 6. Benchmarked recall, latency, and resource usage.
• 7. Deployed the new engine with a feature flag and rolled out to production.

Доказательства

The operational cost grew with it: /api/search averaged 4,034 ms in production. The 300 ms skill prefetch almost never won.

Источник

We moved tokenization into our own code, applied the same way at index time and query time.

Источник

The tokenizer is a single pass over character codes that emits two ordered streams.

Источник