Catching LLM Hallucinations with an In-Cluster Judge Model

Description and use

The article describes a method for detecting LLM hallucinations using a second model running locally in the cluster. This avoids data leakage to third-party providers and reduces costs while maintaining control over response quality.

Problem

Hosted LLM providers often generate unsourced, precise figures (hallucinations) that appear authoritative. Using a hosted model to judge these outputs creates data privacy risks by sending all output to a third party and increases costs.

Outcome

The workflow successfully detects hallucinated figures by using a local, smaller model to judge the output of a larger hosted model. This approach maintains data privacy by keeping the judgment process in-house and reduces costs compared to using a hosted judge.

Workflow

• Deploy a primary generative model via a hosted provider (e.g., OpenAI).
• Set up a local inference server (e.g., Ollama) in the cluster with a smaller judge model (e.g., qwen2.5:3b).
• Configure an API gateway (e.g., agentgateway) to intercept model responses.
• Implement a webhook in the gateway that sends the response to the local judge model.
• Define a rubric for the judge to detect 'unsourced precision' or specific hallucinations.
• Score the response locally; if it fails the check, handle the error or flag it before returning to the user.
• Log the judge's verdict for compliance and audit trails.

Evidence

The second model has no way of knowing whether 30% is true; it never sees the question, and it has no sources. What it judges is whether the answer had any business stating that figure at all.

Source

I ran this with qwen2.5:3b. I started smaller and had to go up.

Source