A Digest Quality Score That Knows What It Can't Prove

Описание и применение

В GistMyMail добавлен детерминированный оценщик качества дайджеста, который проверяет покрытие источников, фактическую обоснованность, действие, дублирование и полноту информации, выдавая отчёт с предупреждениями.

Задача

AI‑generated email digests can present claims that are not supported by the source, leading to false certainty.

Результат

The evaluator provides deterministic, model‑agnostic quality metrics for AI‑generated digests, exposing gaps in source coverage and factual grounding without relying on an LLM judge.

Рабочий процесс

• 1. Generate a digest with GistMyMail.
• 2. The evaluator runs after generation, using the fetched messages and evidence.
• 3. It computes component scores (coverage, factual grounding, actionability, duplicate content, information completeness) and normalizes weights.
• 4. It produces a quality report with rubric version, component scores, counts, statuses, warnings, and source references.
• 5. The report is persisted with the digest and returned by the digest API.
• 6. Reviewers inspect the report via Developer Tools → Network → POST /api/digest/generate → digest.qualityReport.

Доказательства

GistMyMail's v1 quality evaluator makes several checks explicit and repeatable.

Источник

The evaluator changes described below are currently in local working-tree changes and are not yet on the public GitHub branch.

Источник

The evaluator runs after digest generation using the messages and evidence already fetched.

Источник

The report is persisted with the digest and returned by the existing digest API.

Источник