In‑Place Android Screen Translator (ALST)

Description and use

ALST is an Android app that translates any screen in real‑time, overlaying translated text directly over the original content without leaving the app. It uses Gemini 3.6 Flash for cloud translation and ML Kit for offline mode, manages memory leaks, and integrates with system UI.

Problem

Mobile users struggle with language barriers because existing screen‑translation tools require screenshots, freeze screens, or overlay blocking text cards.

Outcome

ALST delivers seamless, real‑time translation directly over app content, improving contextual accuracy with Gemini and maintaining 100% offline capability via ML Kit, all while preventing memory leaks on Android 14+.

Workflow

• 1. Capture screen frames via MediaProjection and ImageReader
• 2. Send bitmap to Gemini 3.6 Flash (cloud) or ML Kit OCR (offline)
• 3. Receive translated text and bounding boxes
• 4. Scale coordinates to device density and insets
• 5. Render glassmorphic overlay cards with Compose Canvas
• 6. Provide Quick Settings tile and floating FAB for quick activation

Evidence

ALST is built following Clean Architecture and MVVM/MVI design patterns.

Source

Gemini extracts the text, understands local context/idioms, and returns both the translated text and exact bounding box coordinates (Rect).

Source

Capturing raw 1440p / 4K screen frames produces bitmaps that consume over 15MB–20MB of RAM per frame.

Source