The check got better and got a new way of being wrong

Описание и применение

Обновленная система проверки заметок AI улучшила некоторые метрики, но привела к новым систематическим ошибкам, особенно в разделах психического здоровья, что может затруднить работу врачей.

Задача

Evaluating AI‑written clinical notes against transcripts revealed that the Note Check engine both improved on some metrics and introduced new, systematic errors, especially in behavioral‑health contexts.

Результат

Engine upgrade reduced false positives in negation rules but introduced systematic Unsupported verdicts due to mis‑labeling of section placement, leading to potential clinician confusion and wasted time.

Рабочий процесс

• 1. Re‑run previous installment’s 36 note pairs with new engine to establish baseline.
• 2. Compare statement counts and verdicts between notecheck‑1 and notecheck‑6.
• 3. Identify changes in Supported, Unsupported, and rule‑flag counts.
• 4. Analyze new Unsupported verdicts for patterns (e.g., section placement errors).
• 5. Document examples where engine is correct versus wrong.
• 6. Summarize impact on clinical workflow and potential clinician time loss.

Доказательства

Between installment 01 and now, the Note Check engine went from notecheck‑1 to notecheck‑6.

Источник

Thirty‑five Unsupported verdicts where there were three. Four of them are wrong, and all four are wrong the same way.

Источник

The check is wrong. Both numbers and the direction of change are in the transcript verbatim.

Источник