Implementing Conversational Search with Manticore and LLMs

Description and use

The article details implementing conversational search using Manticore Search and LLMs. Traditional search returns lists but lacks explanatory power. The proposed solution uses hybrid search for retrieval and an LLM to generate context-based answers, maintaining conversation history. Manticore achieved top scores in a conversational search quality benchmark.

Problem

Traditional full-text and vector search return lists of results but fail to handle complex, multi-turn user queries that require explanation, comparison, or refinement of results based on context.

Outcome

Manticore achieved top scores on a conversational search quality benchmark (Hit@3: 0.3650, Hit@5: 0.4250, Hit@10: 0.5250, MRR: 0.2790) using 200 deterministic shopping queries. The system successfully links LLM answers to specific product sources, ensuring verifiability.

Workflow

• Define a Conversational Search model using CREATE CHAT MODEL, specifying the LLM provider (e.g., OpenRouter), timeout, retrieval limit, max document length, and a system prompt enforcing context-only answers.
• Execute the initial query using CALL CHAT, providing the user query, target table, model name, conversation UUID, and vector field.
• The system uses conversation history to generate a complete search query, performs vector search, retrieves documents, and passes them to the LLM.
• The LLM generates an answer based on retrieved context, citing source IDs.
• Handle follow-up queries by calling CALL CHAT with the same conversation UUID, allowing the system to use history to resolve ambiguous references.
• Access the same functionality via the JSON API by sending a chat object to the /search endpoint.

Evidence

In the current run, Manticore scored 0.3650 Hit@3, 0.4250 Hit@5, 0.5250 Hit@10, and 0.2790 MRR — the best result on each of those metrics among the engines tested.

Source

Google reported in May 2026 that AI Mode had surpassed one billion monthly users.

Source