Build, Adopt, Build Again: Seven Months of Search in a Personal AI Tool

说明与用途

将 Open Walnut 的搜索迁移到自定义分词器和 BM25 引擎，将延迟降低到 <200 ms，并提高了多语言和代码密集型数据的召回率。

问题

The original search engine in Open Walnut was inadequate for mixed-language, code-heavy data, causing poor recall, high latency, and extensive maintenance.

结果

The migration to a custom tokenizer and BM25 engine reduced average search latency from ~4 s to <200 ms, eliminated the need for external vector services, and improved recall for mixed-language and code‑identifier queries.

工作流

• 1. Built a golden‑set harness with real queries and assertions.
• 2. Evaluated the existing QMD engine to establish a baseline.
• 3. Designed a custom tokenizer emitting original and sub‑token streams.
• 4. Implemented the tokenizer at index and query time using SQLite FTS5.
• 5. Replaced the library with the homegrown engine and removed external processes.
• 6. Benchmarked recall, latency, and resource usage.
• 7. Deployed the new engine with a feature flag and rolled out to production.

证据

The operational cost grew with it: /api/search averaged 4,034 ms in production. The 300 ms skill prefetch almost never won.

来源

We moved tokenization into our own code, applied the same way at index time and query time.

来源

The tokenizer is a single pass over character codes that emits two ordered streams.

来源