AI on Edge: One forward pass, one letter: a decision model that runs on a laptop

Description and use

A Gemma 4 E2B model with a LoRA adapter, trained on 7 public datasets, runs locally on a laptop, outputs calibrated confidence scores for intent classification in 85 ms, enabling strict gating and privacy compliance.

Problem

Need a lightweight, calibrated decision model that can run locally on a laptop to provide confidence scores for voice bot intents without generating text, ensuring strict gating and privacy compliance.

Outcome

On 68,000 test rows, calibration error ranged 0.006–0.04; median latency ~85 ms (p95 117 ms). For CLINC150, 90 % of cases handled at ≥0.8 confidence with 98.3 % accuracy; for MASSIVE intent, 79 % handled with 95.4 % accuracy; for BANKING77, 74 % handled with 92.3 % accuracy.

Workflow

• 1. Fine‑tune Gemma 4 E2B (4‑bit) with a 52 MB LoRA adapter on 7 public datasets plus synthetic behaviour rows.
• 2. Train for ~1 hour, peak memory ~5.7 GB.
• 3. At inference, prompt the model once, read raw log‑probabilities for option letters, convert to calibrated probabilities using a temperature fitted on validation.
• 4. Apply a threshold (e.g., 80 %) to decide whether to act or hand off to a human.
• 5. Combine with identity and scoped permission checks in an agent gateway: if permission denied, stop; else if confidence < threshold, send to human; else proceed.

Evidence

The model is Gemma 4 E2B (4-bit) with a LoRA adapter, rank 16. The adapter is 52 MB. Training peaked at about 5.7 GB of memory and took about an hour.

Source

On about 68,000 test rows, the calibration error was between 0.006 and 0.04 on most tasks. Median latency was about 85 ms per decision (p95 117 ms).

Source

The server I use only returns the top 11 log‑probabilities, so a 13‑option list would silently lose probability mass.

Source