Making LLM ratings auditable: quotes, limitations, and "can't judge" instead of zero

Description and use

LLM ratings now include verifiable quotes, limitations, and a transparent "can't judge" state, reducing overconfidence and improving trust.

Problem

How to display LLM-generated ratings without them becoming confident-sounding noise.

Outcome

The system now presents LLM ratings with verifiable evidence, clear limitations, and a transparent handling of unknowns, reducing overconfidence and improving user trust.

Workflow

• 1. Collect up to 20 most recent articles with readable text per account, limiting to first 6,000 characters each.
• 2. Rate each of five dimensions (depth of argument, source transparency, information gain, clarity, caution in stating opinions) on a 1–5 scale.
• 3. Attach a short explanation, a limitations paragraph, and quotes from specific articles (with IDs) as evidence for each dimension.
• 4. Verify each quote against stored article text; remove mismatches.
• 5. If a dimension lacks solid evidence after verification, display "can't judge" instead of zero.
• 6. Show provenance: number of articles used, model name, rubric version, analysis date, and a disclaimer that the score is an AI analysis.
• 7. Provide a rule-based similarity list at the bottom with similarity percentage and shared category, labeled as not quality recommendations.

Evidence

Each account is rated on up to its 20 most recent articles with readable text, using at most the first 6,000 characters of each.

Source

If that leaves a dimension without solid evidence, it's displayed as "can't judge", with a note that the quotes failed verification and the item needs re-analysis or a human review.

Source

Each profile shows how many articles it's based on, the model name, a rubric version string, and the analysis date.

Source