Running an LLM fully offline on Android with Kotlin coroutines

Description and use

The article details embedding a 0.5B‑parameter, 4‑bit quantized LLM into an offline Android app using Kotlin coroutines, addressing thread safety, cancellation, and UTF‑8 streaming.

Problem

Integrating a 0.5B‑parameter, 4‑bit quantized LLM into a native Android app while preserving a Kotlin‑style API, thread safety, cancellation, and correct UTF‑8 streaming.

Outcome

The offline Android app can load a 490 MB 4‑bit quantized model, stream responses with correct UTF‑8, and safely cancel generation, all without GPU, server, or API key.

Workflow

• Load .gguf model file via LlamaModel.load
• Create a new session with newSessionDispatcher() that uses a single‑thread executor
• Call session.chat(listOf(...)).collect{piece->print(piece)} to stream response
• Cancel the coroutine to stop generation
• Use the UTF‑8 streaming decoder to assemble tokens into valid text

Evidence

A 0.5B-parameter chat model, 4-bit quantized, is about 490 MB.

Source

The fix is boring, and that's the point: each session owns one dedicated thread.

Source

Because the generation is a flow { } builder, cancellation comes almost for free.

Source

A model token is a piece of UTF-8, not a character.

Source