A Digest Quality Score That Knows What It Can't Prove

Description and use

GistMyMail now includes a deterministic quality evaluator for AI‑generated digests, checking source coverage, factual grounding, actionability, duplicate content, and information completeness, and attaching a detailed report to each digest.

Problem

AI‑generated email digests can present claims that are not supported by the source, leading to false certainty.

Outcome

The evaluator provides deterministic, model‑agnostic quality metrics for AI‑generated digests, exposing gaps in source coverage and factual grounding without relying on an LLM judge.

Workflow

• 1. Generate a digest with GistMyMail.
• 2. The evaluator runs after generation, using the fetched messages and evidence.
• 3. It computes component scores (coverage, factual grounding, actionability, duplicate content, information completeness) and normalizes weights.
• 4. It produces a quality report with rubric version, component scores, counts, statuses, warnings, and source references.
• 5. The report is persisted with the digest and returned by the digest API.
• 6. Reviewers inspect the report via Developer Tools → Network → POST /api/digest/generate → digest.qualityReport.

Evidence

GistMyMail's v1 quality evaluator makes several checks explicit and repeatable.

Source

The evaluator changes described below are currently in local working-tree changes and are not yet on the public GitHub branch.

Source

The evaluator runs after digest generation using the messages and evidence already fetched.

Source

The report is persisted with the digest and returned by the existing digest API.

Source