Invisible characters in 2,392 Android apps' translations: Weblate's zero-width-space fix shows up in what ships

Описание и применение

В Android‑приложениях часто появляются невидимые символы, такие как нулевые ширины, мягкие дефисы и BOM, что приводит к ошибкам поиска и сравнения строк. Исследование показало, что репозитории, использующие Weblate, содержат в среднем в четыре раза меньше нулевых ширин, но не очищают другие типы символов. В результате был создан новый предупреждающий флаг l10n‑guard, который обнаруживает такие символы в CI.

Задача

Invisible formatting characters such as zero‑width spaces, soft hyphens, byte‑order marks and bidirectional controls are introduced into Android app translations through copy‑and‑paste from translation tools and web pages, breaking search, string comparison and line wrapping.

Результат

Weblate repositories ship about a quarter as many stray zero‑width spaces per million units compared to non‑Weblate repos (17 vs 70), but they are not cleaner for other control characters (531 vs 303). The study also revealed that many left‑to‑right marks in English pseudo‑locales were due to an unaccounted en‑XC locale.

Рабочий процесс

• 1. Identify all F‑Droid repositories and their translation catalogs.
• 2. Classify repositories by whether they use Weblate (493) or not (1,899).
• 3. Count invisible characters that are present in translations but absent in the English source, separating zero‑width spaces (U+200B) and other control‑class characters.
• 4. Compute the ratio of extra characters per million translated units for Weblate and non‑Weblate repos.
• 5. Analyze the distribution of soft hyphens, byte‑order marks, and other stray characters.
• 6. Publish findings and create a new l10n‑guard warning to flag these issues in CI.

Доказательства

Translations pick up invisible characters: zero-width spaces, soft hyphens, byte-order marks and bidi controls.

Источник

Weblate already deals with one of them. Its RemoveZeroSpace autofix deletes a U+200B that the source string doesn't have (Khmer is exempt).

Источник

Weblate repositories ship about a quarter as many stray zero-width spaces, but they are not cleaner on the characters Weblate leaves alone.

Источник

These findings shaped a new invisible-char warning in l10n-guard 0.15.0, MV2's open-source (Apache-2.0) translation checker and GitHub Action.

Источник