The check got better and got a new way of being wrong

Description and use

The upgraded Note Check engine improved some metrics but introduced systematic Unsupported verdicts, especially in behavioral‑health sections, potentially confusing clinicians.

Problem

Evaluating AI‑written clinical notes against transcripts revealed that the Note Check engine both improved on some metrics and introduced new, systematic errors, especially in behavioral‑health contexts.

Outcome

Engine upgrade reduced false positives in negation rules but introduced systematic Unsupported verdicts due to mis‑labeling of section placement, leading to potential clinician confusion and wasted time.

Workflow

• 1. Re‑run previous installment’s 36 note pairs with new engine to establish baseline.
• 2. Compare statement counts and verdicts between notecheck‑1 and notecheck‑6.
• 3. Identify changes in Supported, Unsupported, and rule‑flag counts.
• 4. Analyze new Unsupported verdicts for patterns (e.g., section placement errors).
• 5. Document examples where engine is correct versus wrong.
• 6. Summarize impact on clinical workflow and potential clinician time loss.

Evidence

Between installment 01 and now, the Note Check engine went from notecheck‑1 to notecheck‑6.

Source

Thirty‑five Unsupported verdicts where there were three. Four of them are wrong, and all four are wrong the same way.

Source

The check is wrong. Both numbers and the direction of change are in the transcript verbatim.

Source