One Packet to Tune, Two to Check: Tuning Scribe Without Fooling Yourself

Description and use

Optimizing the Scribe handwriting‑reading model on long questionnaire packets: achieving 96 % exact accuracy and reducing processing time to 47 min by fixing input bugs, correcting the scorer, and adding a second read.

Problem

How to tune a document‑reading pipeline on a single long‑form packet while ensuring gains are real and not artifacts of the scorer or data leakage.

Outcome

The tuned pipeline achieved a 2.8 percentage point increase in exact accuracy and a 2.3 percentage point increase in accepted‑answer accuracy, while reducing wrong answers escaping review from 11 to 1 and cutting processing time from 70 to 47 minutes.

Workflow

• 1. Built a unified packet schema with 177 answer fields and 89 checks.
• 2. Created three filled packets: one for tuning, two for checking.
• 3. Generated answer keys via dual AI reads and manual review, achieving 99.2% agreement.
• 4. Ran initial tuning on the hardest packet, measuring gains on check packets.
• 5. Identified scorer bias by comparing prompts with/without “reading skills” notes; corrected scorer to use a fair convention.
• 6. Debugged model inputs: fixed cropping failures and column misalignment in tick grids.
• 7. Removed tick fields, keeping only handwritten scores to reduce noise and review load.
• 8. Added a second, smaller model for re‑reading; kept only the re‑read of the same image to reduce false alarms.
• 9. Optimized speed: batch‑read 4 answers per page, capped model notes, and implemented prefix caching.
• 10. Validated each change with paired runs on tuning and check packets to ensure no regression.

Evidence

On 531 checked answers across the three packets, the final configuration ended up with: 96% read exactly right, up from 93% (94.4% to 96.0% on the two check packets alone).

Source

The first experiment compared prompts with and without short "reading skills": form-agnostic notes on how pen marks work… The skills version looked 1.5 points better. It wasn't.

Source

I built a debugging view that shows, for every wrong answer, the image the model read next to where the answer actually is on the page.

Source

On the tuning packet alone, three rounds of changes took exact accuracy from 85.6% to 89.1% and accepted-answer accuracy from 90.6% to 98.6%.

Source

Part 1's best idea was a second, different model re-reading each answer… 92% were false alarms (55 of 60).

Source