How to Test Localization and Device Compatibility at Scale

Description and use

Separating locale and device matrices, parameterising key scenarios, and asserting format correctness with ICU/CLDR reduced regression time to one day and achieved 70 % translation coverage.

Problem

Testing all locale and device combinations (e.g., 27 localizations × 20+ devices × 4 browsers = 2,160 combinations) is infeasible; teams rarely run the full matrix and miss many defects.

Outcome

By separating locale and device axes and focusing on high‑value scenarios, regression time dropped from 2.5 weeks to 1 day, smoke testing from 7 days to 1 day, and 70% of translations gained automated coverage.

Workflow

• Sweep locales on a single reference runtime (browser, OS, screen size).
• Sweep devices on a single reference locale (runtime).
• Select a handful of intersection cells (e.g., longest‑expanding language on smallest screen, RTL locale on oldest OS).
• Parameterise high‑value scenarios (signup, checkout, money display, date display) across all locales.
• Assert on format correctness, missing‑key markers, and clipped text using ICU/CLDR data.
• Run locale dimension in CI as a matrix rather than a full suite.

Evidence

27 localizations, 20+ devices, 4 browsers. Multiply them and you get 2,160 combinations.

Source

Regression down from 2.5 weeks to 1 day, smoke from 7 days to 1 day, and 70% of translations covered by automated tests.

Source

Do not run full regression per locale. Parameterise a small, high‑value set of scenarios — signup, checkout, the one screen that displays money, the one that displays dates — across every locale, and run everything else once.

Source