Catching LLM Hallucinations with an In-Cluster Judge Model

Описание и применение

Статья описывает метод обнаружения галлюцинаций LLM с помощью второй модели, работающей локально в кластере. Это позволяет избежать утечки данных к сторонним провайдерам и снизить затраты, сохраняя при этом контроль над качеством ответов.

Задача

Hosted LLM providers often generate unsourced, precise figures (hallucinations) that appear authoritative. Using a hosted model to judge these outputs creates data privacy risks by sending all output to a third party and increases costs.

Результат

The workflow successfully detects hallucinated figures by using a local, smaller model to judge the output of a larger hosted model. This approach maintains data privacy by keeping the judgment process in-house and reduces costs compared to using a hosted judge.

Рабочий процесс

• Deploy a primary generative model via a hosted provider (e.g., OpenAI).
• Set up a local inference server (e.g., Ollama) in the cluster with a smaller judge model (e.g., qwen2.5:3b).
• Configure an API gateway (e.g., agentgateway) to intercept model responses.
• Implement a webhook in the gateway that sends the response to the local judge model.
• Define a rubric for the judge to detect 'unsourced precision' or specific hallucinations.
• Score the response locally; if it fails the check, handle the error or flag it before returning to the user.
• Log the judge's verdict for compliance and audit trails.

Доказательства

The second model has no way of knowing whether 30% is true; it never sees the question, and it has no sources. What it judges is whether the answer had any business stating that figure at all.

Источник

I ran this with qwen2.5:3b. I started smaller and had to go up.

Источник