Making LLM ratings auditable: quotes, limitations, and "can't judge" instead of zero

说明与用途

LLM 评分现在包含可验证的引用、限制和透明的 "无法判断" 状态，降低过度自信并提升信任度。

问题

How to display LLM-generated ratings without them becoming confident-sounding noise.

结果

The system now presents LLM ratings with verifiable evidence, clear limitations, and a transparent handling of unknowns, reducing overconfidence and improving user trust.

工作流

• 1. Collect up to 20 most recent articles with readable text per account, limiting to first 6,000 characters each.
• 2. Rate each of five dimensions (depth of argument, source transparency, information gain, clarity, caution in stating opinions) on a 1–5 scale.
• 3. Attach a short explanation, a limitations paragraph, and quotes from specific articles (with IDs) as evidence for each dimension.
• 4. Verify each quote against stored article text; remove mismatches.
• 5. If a dimension lacks solid evidence after verification, display "can't judge" instead of zero.
• 6. Show provenance: number of articles used, model name, rubric version, analysis date, and a disclaimer that the score is an AI analysis.
• 7. Provide a rule-based similarity list at the bottom with similarity percentage and shared category, labeled as not quality recommendations.

证据

Each account is rated on up to its 20 most recent articles with readable text, using at most the first 6,000 characters of each.

来源

If that leaves a dimension without solid evidence, it's displayed as "can't judge", with a note that the quotes failed verification and the item needs re-analysis or a human review.

来源

Each profile shows how many articles it's based on, the model name, a rubric version string, and the analysis date.

来源