AI on Edge: One forward pass, one letter: a decision model that runs on a laptop

Описание и применение

Модель на базе Gemma 4 E2B с LoRA‑адаптером, обученная на 7 публичных датасетах, работает локально на ноутбуке, выдаёт точные вероятности для выбора интента в 85 мс, позволяя строгий контроль доступа и соблюдение конфиденциальности.

Задача

Need a lightweight, calibrated decision model that can run locally on a laptop to provide confidence scores for voice bot intents without generating text, ensuring strict gating and privacy compliance.

Результат

On 68,000 test rows, calibration error ranged 0.006–0.04; median latency ~85 ms (p95 117 ms). For CLINC150, 90 % of cases handled at ≥0.8 confidence with 98.3 % accuracy; for MASSIVE intent, 79 % handled with 95.4 % accuracy; for BANKING77, 74 % handled with 92.3 % accuracy.

Рабочий процесс

• 1. Fine‑tune Gemma 4 E2B (4‑bit) with a 52 MB LoRA adapter on 7 public datasets plus synthetic behaviour rows.
• 2. Train for ~1 hour, peak memory ~5.7 GB.
• 3. At inference, prompt the model once, read raw log‑probabilities for option letters, convert to calibrated probabilities using a temperature fitted on validation.
• 4. Apply a threshold (e.g., 80 %) to decide whether to act or hand off to a human.
• 5. Combine with identity and scoped permission checks in an agent gateway: if permission denied, stop; else if confidence < threshold, send to human; else proceed.

Доказательства

The model is Gemma 4 E2B (4-bit) with a LoRA adapter, rank 16. The adapter is 52 MB. Training peaked at about 5.7 GB of memory and took about an hour.

Источник

On about 68,000 test rows, the calibration error was between 0.006 and 0.04 on most tasks. Median latency was about 85 ms per decision (p95 117 ms).

Источник

The server I use only returns the top 11 log‑probabilities, so a 13‑option list would silently lose probability mass.

Источник