AI on Edge: One forward pass, one letter: a decision model that runs on a laptop

说明与用途

基于Gemma 4 E2B的LoRA适配器模型，在笔记本上本地运行，提供85 ms的校准置信度评分，实现严格的门控和隐私合规。

问题

Need a lightweight, calibrated decision model that can run locally on a laptop to provide confidence scores for voice bot intents without generating text, ensuring strict gating and privacy compliance.

结果

On 68,000 test rows, calibration error ranged 0.006–0.04; median latency ~85 ms (p95 117 ms). For CLINC150, 90 % of cases handled at ≥0.8 confidence with 98.3 % accuracy; for MASSIVE intent, 79 % handled with 95.4 % accuracy; for BANKING77, 74 % handled with 92.3 % accuracy.

工作流

• 1. Fine‑tune Gemma 4 E2B (4‑bit) with a 52 MB LoRA adapter on 7 public datasets plus synthetic behaviour rows.
• 2. Train for ~1 hour, peak memory ~5.7 GB.
• 3. At inference, prompt the model once, read raw log‑probabilities for option letters, convert to calibrated probabilities using a temperature fitted on validation.
• 4. Apply a threshold (e.g., 80 %) to decide whether to act or hand off to a human.
• 5. Combine with identity and scoped permission checks in an agent gateway: if permission denied, stop; else if confidence < threshold, send to human; else proceed.

证据

The model is Gemma 4 E2B (4-bit) with a LoRA adapter, rank 16. The adapter is 52 MB. Training peaked at about 5.7 GB of memory and took about an hour.

来源

On about 68,000 test rows, the calibration error was between 0.006 and 0.04 on most tasks. Median latency was about 85 ms per decision (p95 117 ms).

来源

The server I use only returns the top 11 log‑probabilities, so a 13‑option list would silently lose probability mass.

来源