Preparing trustworthy training data for Hanifi Rohingya OCR

Описание и применение

Создана рабочая схема подготовки достоверных обучающих данных для OCR Hanifi Rohingya, включающая единичные линии, манифест с метаданными, нормализацию Unicode, разделение синтетических и реальных изображений и оценку ошибок с помощью расстояния редактирования.

Задача

How to ensure training examples for a Hanifi Rohingya OCR are accurate and trustworthy?

Результат

The workflow establishes a clear pipeline for creating trustworthy OCR training data, separating synthetic from real data, ensuring line‑by‑line alignment, preserving Unicode integrity, and preventing data leakage through document‑based splits.

Рабочий процесс

• Define unit of annotation: one image per transcribed line.
• Create paired image and .gt.txt files following Tesseract tesstrain conventions.
• Maintain a manifest JSON with sample_id, source_document_id, image, label, origin, review_status, and permission fields.
• Preserve Unicode text and display it with dir="rtl" and lang="rhg" in review UI.
• Use a font that supports Hanifi for visual inspection.
• Normalize text with NFC and collapse whitespace before comparison.
• Generate synthetic images from dictionary forms, record the source string, and keep synthetic evaluation separate from real data.
• Split real data by source document before distributing line crops to training or evaluation sets.
• Compute edit distance (CER) on normalized strings to quote error rates.
• Document all steps and decisions for reproducibility.

Доказательства

For a line recognizer, one image should correspond to one transcribed line.

Источник

The .gt.txt file contains the exact text visible in that crop.

Источник

A practical next step is to keep a manifest alongside the pairs.

Источник

The document ID matters when we create an evaluation split.

Источник

My evaluation helper uses edit distance between normalized Unicode strings.

Источник