Closing the KNN rescoring gap between columnar and row-wise storage

Description and use

In Manticore Search, default KNN rescoring is slower with columnar storage than row-wise. Switching to memory-mapped columnar access reduces the throughput gap to 85–92% of row-wise performance while preserving out‑of‑core efficiency.

Problem

Default KNN rescoring in Manticore Search makes columnar vector access slower than row-wise storage, creating a performance gap of 2.80x–3.53x in KNN throughput.

Outcome

Memory-mapped columnar access reduces the KNN rescoring gap from 2.80x–3.53x to 1.08x–1.15x, achieving 85–92% of row-wise throughput while maintaining out-of-core performance.

Workflow

• Measure KNN throughput on DBpedia dataset with row-wise and columnar storage.
• Identify rescoring bottleneck caused by reusable buffer.
• Implement memory-mapped columnar access to provide stable pointers.
• Batch rescoring: sort candidates, collect pointers, prefetch data, calculate distances in batches.
• Re-run benchmarks to compare throughput.
• Repeat with out-of-core taxi dataset to evaluate performance under memory pressure.
• Compare cold and hot run times for file vs mmap access.

Evidence

With the previous columnar access path, that step was much slower than with row-wise storage. In our DBpedia benchmark, row-wise storage delivered 2.80x–3.53x the KNN throughput.

Source

We changed columnar access to use memory mapping. This gives vectors stable addresses while still allowing the operating system to load and evict file pages on demand. The result is 2.57x–3.13x higher KNN throughput, reaching 85–92% of row-wise performance.

Source