Better Vector Search for Long Documents: Chunking Inside Manticore Search

Описание и применение

Ментикара автоматически разбивает длинные документы на чанки при вставке, улучшая точность поиска с 55,1 % до 83,3 % recall@5 и повышая MRR, при умеренных накладных расходах на память и время загрузки.

Задача

Embedding models with limited token windows truncate long documents, causing loss of content and poor search recall.

Результат

Using Manticore’s chunk_strategy increased recall@5 from 55.1% to 83.3% and MRR from 0.44 to 0.70 on a 189‑page manual, with only ~2.5× RAM and ~4× ingest time overhead.

Рабочий процесс

• Create a table with a float_vector_array column and set chunk_strategy (e.g., 'sentence'), max_tokens, and overlap_tokens.
• Insert documents; Manticore splits each document into chunks, generates embeddings for each chunk, and stores them in the array column.
• Run vector queries; Manticore searches all chunks and returns the document once, reporting the nearest chunk distance.
• Optionally adjust chunking parameters (max_tokens, overlap_tokens, max_chunks) to tune performance and recall.
• Monitor recall@5 and MRR to validate improvements.

Доказательства

Recall@5 went from 55.1% → 83.3% and MRR from 0.44 → 0.70, at ~2.5× the RAM and ~4× the ingest time.

Источник