Paperless-ngx migrated from Whoosh to Tantivy while preserving query syntax

Description and use

Paperless-ngx migrated from Whoosh to Tantivy to improve indexing speed and reduce disk usage. To preserve user experience, they built a translation layer that converts Whoosh query syntax to Tantivy queries, ensuring existing saved searches and user habits remained intact.

Problem

Paperless-ngx needed to improve indexing speed and reduce on-disk index size for its full-text search, but wanted to avoid breaking existing user queries and saved searches that relied on the Whoosh query language.

Outcome

Paperless-ngx successfully migrated to Tantivy for better performance and smaller index sizes. By building a translation layer for the query language, they preserved the user experience, allowing existing queries and saved searches to function without modification.

Workflow

• Identify performance bottlenecks in the existing pure-Python search engine (Whoosh).
• Select a high-performance alternative engine (Tantivy) with Python bindings.
• Analyze the existing query language syntax used by end-users.
• Develop a translation layer or companion library to parse the old query syntax and convert it to the new engine's query format.
• Implement the new backend while maintaining the original query interface for users.
• Release the migration (v3.0) ensuring saved searches and muscle-memory queries continue to work without user intervention.

Evidence

In its 3.0 release, the team replaced the Whoosh backend with tantivy — a Rust search engine with Python bindings — for indexing speed and a smaller on-disk index.

Source

A companion library was built specifically to parse the Whoosh query language and translate it into tantivy queries, so that every saved search and every muscle-memory query users had written over the years kept working.

Source