Kev gives the same answer every time. Until you batch it.

说明与用途

研究表明，Kev‑4B 4‑bit 在单请求或小批量请求时输出确定，但在大批量时概率漂移，导致少数答案变化。若在留出数据上设定阈值，75% 自动化率仍然有效。

问题

Determining whether a quantized decision model (Kev-4B 4-bit) produces deterministic outputs under different load conditions and how confidence thresholds affect automation.

结果

Kev-4B is deterministic for single‑request or small‑batch runs but shows probability drift in larger batches, leading to minimal top‑choice changes; the 75 % automation claim holds when thresholds are chosen on held‑out data. Julia‑1 performed poorly on the independent suite.

工作流

• 1. Run Kev-4B 4‑bit on 866 questions once (baseline).
• 2. Re‑run the same 866 questions three more times, one request at a time, compare probabilities to baseline.
• 3. Repeat with concurrent batches of 2, 4, 8, and 16 requests, recording probabilities and decisions.
• 4. Compare batched runs to each other to assess consistency.
• 5. Evaluate decision changes and compute accuracy for each batch size.
• 6. Re‑estimate the 75 % automation threshold using a held‑out split of the data.
• 7. Repeat the same tests for Julia‑1 and other models for comparison.

证据

On 28 September we sent Kev-4B 4-bit the same 866 questions three more times, one request at a time, and compared every answer with the original run from 26 September.

来源

Grouped by the size of the batch each request actually ran in, the line is sharp. In the first run at 8, none of the 116 requests that ran alone or in pairs changed, while 311 of the 735 that ran in batches of seven did.

来源

The effect on decisions was small. Across the three batched runs, 2 of 2,598 answers changed their top choice, and one of those turned a right answer into a wrong one.

来源

We should have explained this the first time. "Automatable at 5% error" doesn't use the 90% threshold.

来源