Paperless-ngx migrated from Whoosh to Tantivy while preserving query syntax

说明与用途

Paperless-ngx 从 Whoosh 迁移到 Tantivy 以提高索引速度并减小磁盘占用。为了保留用户体验，他们构建了一个翻译层，将 Whoosh 查询语法转换为 Tantivy 查询，确保现有的保存搜索和用户习惯保持不变。

问题

Paperless-ngx needed to improve indexing speed and reduce on-disk index size for its full-text search, but wanted to avoid breaking existing user queries and saved searches that relied on the Whoosh query language.

结果

Paperless-ngx successfully migrated to Tantivy for better performance and smaller index sizes. By building a translation layer for the query language, they preserved the user experience, allowing existing queries and saved searches to function without modification.

工作流

• Identify performance bottlenecks in the existing pure-Python search engine (Whoosh).
• Select a high-performance alternative engine (Tantivy) with Python bindings.
• Analyze the existing query language syntax used by end-users.
• Develop a translation layer or companion library to parse the old query syntax and convert it to the new engine's query format.
• Implement the new backend while maintaining the original query interface for users.
• Release the migration (v3.0) ensuring saved searches and muscle-memory queries continue to work without user intervention.

证据

In its 3.0 release, the team replaced the Whoosh backend with tantivy — a Rust search engine with Python bindings — for indexing speed and a smaller on-disk index.

来源

A companion library was built specifically to parse the Whoosh query language and translate it into tantivy queries, so that every saved search and every muscle-memory query users had written over the years kept working.

来源