Preparing trustworthy training data for Hanifi Rohingya OCR

说明与用途

为 Hanifi Rohingya OCR 提供了可信训练数据的工作流程，包括逐行标注、元数据清单、Unicode 规范化、合成与真实图像分离以及 CER 评估。

问题

How to ensure training examples for a Hanifi Rohingya OCR are accurate and trustworthy?

结果

The workflow establishes a clear pipeline for creating trustworthy OCR training data, separating synthetic from real data, ensuring line‑by‑line alignment, preserving Unicode integrity, and preventing data leakage through document‑based splits.

工作流

• Define unit of annotation: one image per transcribed line.
• Create paired image and .gt.txt files following Tesseract tesstrain conventions.
• Maintain a manifest JSON with sample_id, source_document_id, image, label, origin, review_status, and permission fields.
• Preserve Unicode text and display it with dir="rtl" and lang="rhg" in review UI.
• Use a font that supports Hanifi for visual inspection.
• Normalize text with NFC and collapse whitespace before comparison.
• Generate synthetic images from dictionary forms, record the source string, and keep synthetic evaluation separate from real data.
• Split real data by source document before distributing line crops to training or evaluation sets.
• Compute edit distance (CER) on normalized strings to quote error rates.
• Document all steps and decisions for reproducibility.

证据

For a line recognizer, one image should correspond to one transcribed line.

来源

The .gt.txt file contains the exact text visible in that crop.

来源

A practical next step is to keep a manifest alongside the pairs.

来源

The document ID matters when we create an evaluation split.

来源

My evaluation helper uses edit distance between normalized Unicode strings.

来源