One Packet to Tune, Two to Check: Tuning Scribe Without Fooling Yourself

Описание и применение

Оптимизация модели распознавания рукописного текста Scribe на длинных формах: улучшение точности до 96 % и сокращение времени обработки до 47 минут за счёт устранения ошибок в скриншотах, коррекции оценщика и добавления второго чтения.

Задача

How to tune a document‑reading pipeline on a single long‑form packet while ensuring gains are real and not artifacts of the scorer or data leakage.

Результат

The tuned pipeline achieved a 2.8 percentage point increase in exact accuracy and a 2.3 percentage point increase in accepted‑answer accuracy, while reducing wrong answers escaping review from 11 to 1 and cutting processing time from 70 to 47 minutes.

Рабочий процесс

• 1. Built a unified packet schema with 177 answer fields and 89 checks.
• 2. Created three filled packets: one for tuning, two for checking.
• 3. Generated answer keys via dual AI reads and manual review, achieving 99.2% agreement.
• 4. Ran initial tuning on the hardest packet, measuring gains on check packets.
• 5. Identified scorer bias by comparing prompts with/without “reading skills” notes; corrected scorer to use a fair convention.
• 6. Debugged model inputs: fixed cropping failures and column misalignment in tick grids.
• 7. Removed tick fields, keeping only handwritten scores to reduce noise and review load.
• 8. Added a second, smaller model for re‑reading; kept only the re‑read of the same image to reduce false alarms.
• 9. Optimized speed: batch‑read 4 answers per page, capped model notes, and implemented prefix caching.
• 10. Validated each change with paired runs on tuning and check packets to ensure no regression.

Доказательства

On 531 checked answers across the three packets, the final configuration ended up with: 96% read exactly right, up from 93% (94.4% to 96.0% on the two check packets alone).

Источник

The first experiment compared prompts with and without short "reading skills": form-agnostic notes on how pen marks work… The skills version looked 1.5 points better. It wasn't.

Источник

I built a debugging view that shows, for every wrong answer, the image the model read next to where the answer actually is on the page.

Источник

On the tuning packet alone, three rounds of changes took exact accuracy from 85.6% to 89.1% and accepted-answer accuracy from 90.6% to 98.6%.

Источник

Part 1's best idea was a second, different model re-reading each answer… 92% were false alarms (55 of 60).

Источник