Preparing trustworthy training data for Hanifi Rohingya OCR

Description and use

A workflow for trustworthy training data in Hanifi Rohingya OCR is outlined, covering line‑by‑line annotation, manifest metadata, Unicode preservation, synthetic vs real splits, and CER evaluation.

Problem

How to ensure training examples for a Hanifi Rohingya OCR are accurate and trustworthy?

Outcome

The workflow establishes a clear pipeline for creating trustworthy OCR training data, separating synthetic from real data, ensuring line‑by‑line alignment, preserving Unicode integrity, and preventing data leakage through document‑based splits.

Workflow

• Define unit of annotation: one image per transcribed line.
• Create paired image and .gt.txt files following Tesseract tesstrain conventions.
• Maintain a manifest JSON with sample_id, source_document_id, image, label, origin, review_status, and permission fields.
• Preserve Unicode text and display it with dir="rtl" and lang="rhg" in review UI.
• Use a font that supports Hanifi for visual inspection.
• Normalize text with NFC and collapse whitespace before comparison.
• Generate synthetic images from dictionary forms, record the source string, and keep synthetic evaluation separate from real data.
• Split real data by source document before distributing line crops to training or evaluation sets.
• Compute edit distance (CER) on normalized strings to quote error rates.
• Document all steps and decisions for reproducibility.

Evidence

For a line recognizer, one image should correspond to one transcribed line.

Source

The .gt.txt file contains the exact text visible in that crop.

Source

A practical next step is to keep a manifest alongside the pairs.

Source

The document ID matters when we create an evaluation split.

Source

My evaluation helper uses edit distance between normalized Unicode strings.

Source