Paperless-ngx migrated from Whoosh to Tantivy while preserving query syntax

Описание и применение

Paperless-ngx перешел с Whoosh на Tantivy для повышения скорости индексации и уменьшения размера индекса. Чтобы не ломать пользовательский опыт, команда создала библиотеку-переводчик, которая преобразует запросы в синтаксисе Whoosh в запросы для Tantivy, сохраняя работоспособность сохраненных поисков.

Задача

Paperless-ngx needed to improve indexing speed and reduce on-disk index size for its full-text search, but wanted to avoid breaking existing user queries and saved searches that relied on the Whoosh query language.

Результат

Paperless-ngx successfully migrated to Tantivy for better performance and smaller index sizes. By building a translation layer for the query language, they preserved the user experience, allowing existing queries and saved searches to function without modification.

Рабочий процесс

• Identify performance bottlenecks in the existing pure-Python search engine (Whoosh).
• Select a high-performance alternative engine (Tantivy) with Python bindings.
• Analyze the existing query language syntax used by end-users.
• Develop a translation layer or companion library to parse the old query syntax and convert it to the new engine's query format.
• Implement the new backend while maintaining the original query interface for users.
• Release the migration (v3.0) ensuring saved searches and muscle-memory queries continue to work without user intervention.

Доказательства

In its 3.0 release, the team replaced the Whoosh backend with tantivy — a Rust search engine with Python bindings — for indexing speed and a smaller on-disk index.

Источник

A companion library was built specifically to parse the Whoosh query language and translate it into tantivy queries, so that every saved search and every muscle-memory query users had written over the years kept working.

Источник