Running an LLM fully offline on Android with Kotlin coroutines

说明与用途

本文介绍了如何在离线 Android 应用中集成 0.5B 参数、4‑bit 量化的 LLM，使用 Kotlin 协程实现线程安全、取消生成和正确的 UTF‑8 流式解码。

问题

Integrating a 0.5B‑parameter, 4‑bit quantized LLM into a native Android app while preserving a Kotlin‑style API, thread safety, cancellation, and correct UTF‑8 streaming.

结果

The offline Android app can load a 490 MB 4‑bit quantized model, stream responses with correct UTF‑8, and safely cancel generation, all without GPU, server, or API key.

工作流

• Load .gguf model file via LlamaModel.load
• Create a new session with newSessionDispatcher() that uses a single‑thread executor
• Call session.chat(listOf(...)).collect{piece->print(piece)} to stream response
• Cancel the coroutine to stop generation
• Use the UTF‑8 streaming decoder to assemble tokens into valid text

证据

A 0.5B-parameter chat model, 4-bit quantized, is about 490 MB.

来源

The fix is boring, and that's the point: each session owns one dedicated thread.

来源

Because the generation is a flow { } builder, cancellation comes almost for free.

来源

A model token is a piece of UTF-8, not a character.

来源