One Packet to Tune, Two to Check: Tuning Scribe Without Fooling Yourself

说明与用途

对Scribe手写识别模型在长表单上的优化：通过修复输入错误、校正评分器并添加第二次读取，将准确率提升至96%，处理时间缩短至47分钟。

问题

How to tune a document‑reading pipeline on a single long‑form packet while ensuring gains are real and not artifacts of the scorer or data leakage.

结果

The tuned pipeline achieved a 2.8 percentage point increase in exact accuracy and a 2.3 percentage point increase in accepted‑answer accuracy, while reducing wrong answers escaping review from 11 to 1 and cutting processing time from 70 to 47 minutes.

工作流

• 1. Built a unified packet schema with 177 answer fields and 89 checks.
• 2. Created three filled packets: one for tuning, two for checking.
• 3. Generated answer keys via dual AI reads and manual review, achieving 99.2% agreement.
• 4. Ran initial tuning on the hardest packet, measuring gains on check packets.
• 5. Identified scorer bias by comparing prompts with/without “reading skills” notes; corrected scorer to use a fair convention.
• 6. Debugged model inputs: fixed cropping failures and column misalignment in tick grids.
• 7. Removed tick fields, keeping only handwritten scores to reduce noise and review load.
• 8. Added a second, smaller model for re‑reading; kept only the re‑read of the same image to reduce false alarms.
• 9. Optimized speed: batch‑read 4 answers per page, capped model notes, and implemented prefix caching.
• 10. Validated each change with paired runs on tuning and check packets to ensure no regression.

证据

On 531 checked answers across the three packets, the final configuration ended up with: 96% read exactly right, up from 93% (94.4% to 96.0% on the two check packets alone).

来源

The first experiment compared prompts with and without short "reading skills": form-agnostic notes on how pen marks work… The skills version looked 1.5 points better. It wasn't.

来源

I built a debugging view that shows, for every wrong answer, the image the model read next to where the answer actually is on the page.

来源

On the tuning packet alone, three rounds of changes took exact accuracy from 85.6% to 89.1% and accepted-answer accuracy from 90.6% to 98.6%.

来源

Part 1's best idea was a second, different model re-reading each answer… 92% were false alarms (55 of 60).

来源