Running an LLM fully offline on Android with Kotlin coroutines

Описание и применение

В статье описывается интеграция 0,5‑млн параметров LLM в Android‑приложение без сети, с использованием Kotlin‑корутин, обеспечения потокобезопасности, корректной отмены генерации и правильного декодирования UTF‑8.

Задача

Integrating a 0.5B‑parameter, 4‑bit quantized LLM into a native Android app while preserving a Kotlin‑style API, thread safety, cancellation, and correct UTF‑8 streaming.

Результат

The offline Android app can load a 490 MB 4‑bit quantized model, stream responses with correct UTF‑8, and safely cancel generation, all without GPU, server, or API key.

Рабочий процесс

• Load .gguf model file via LlamaModel.load
• Create a new session with newSessionDispatcher() that uses a single‑thread executor
• Call session.chat(listOf(...)).collect{piece->print(piece)} to stream response
• Cancel the coroutine to stop generation
• Use the UTF‑8 streaming decoder to assemble tokens into valid text

Доказательства

A 0.5B-parameter chat model, 4-bit quantized, is about 490 MB.

Источник

The fix is boring, and that's the point: each session owns one dedicated thread.

Источник

Because the generation is a flow { } builder, cancellation comes almost for free.

Источник

A model token is a piece of UTF-8, not a character.

Источник