Measuring what local AI gets wrong in translated dialogue

Description and use

A Python toolkit checks translated dialogue for timing, overlap, drift, and terminology, revealing failures that require human review before dubbing.

Problem

Local open-weight models produce translated dialogue that may not fit timing constraints or preserve meaning, making automated dubbing unreliable.

Outcome

The toolkit identified that 4 of 12 translations fit timing, 2 needed stretching, 2 were too short, and 4 were too long. Shorter variants from Gemma improved fit for 3 lines, but timing alone did not guarantee meaning preservation.

Workflow

• Translate dialogue with Opus‑MT (English→French) or generate shorter variants with Gemma 3 4B via Ollama
• Run localisation‑measurements verifier to check dialogue fit, timing, overlap, drift, terminology
• Review failed lines and adjust or reject them
• Optionally regenerate shorter variants with Gemma
• Repeat verification until all checks pass

Evidence

Localisation measurements is a Python toolkit with four command-line checks: Estimated dialogue fit inside a time window. Subtitle timing, overlap and reading speed. Latency and accumulated drift budgets. Terminology consistency and figures.

Source

The core tools use only the Python standard library. A separate example uses Opus-MT for English-to-French translation and Gemma 3 4B through local Ollama for shorter variants.

Source

The verifier passes 78 cases checking exit codes and hand-calculated results. The example input can deliberately produce a failing quality-check exit code: that means a measurement found a problem.

Source

Line 008 changes from past to present tense while passing its timing check. All candidates require human review.

Source