Kev gives the same answer every time. Until you batch it.

Описание и применение

Исследование показало, что модель Kev‑4B 4‑бит обеспечивает детерминированные ответы при одиночных запросах и небольших пакетах, но при больших пакетах вероятности дрейфуют, что приводит к редким изменениям в выборе ответа. При правильном выборе порога на отложенных данных 75 % автоматизации остаётся валидным.

Задача

Determining whether a quantized decision model (Kev-4B 4-bit) produces deterministic outputs under different load conditions and how confidence thresholds affect automation.

Результат

Kev-4B is deterministic for single‑request or small‑batch runs but shows probability drift in larger batches, leading to minimal top‑choice changes; the 75 % automation claim holds when thresholds are chosen on held‑out data. Julia‑1 performed poorly on the independent suite.

Рабочий процесс

• 1. Run Kev-4B 4‑bit on 866 questions once (baseline).
• 2. Re‑run the same 866 questions three more times, one request at a time, compare probabilities to baseline.
• 3. Repeat with concurrent batches of 2, 4, 8, and 16 requests, recording probabilities and decisions.
• 4. Compare batched runs to each other to assess consistency.
• 5. Evaluate decision changes and compute accuracy for each batch size.
• 6. Re‑estimate the 75 % automation threshold using a held‑out split of the data.
• 7. Repeat the same tests for Julia‑1 and other models for comparison.

Доказательства

On 28 September we sent Kev-4B 4-bit the same 866 questions three more times, one request at a time, and compared every answer with the original run from 26 September.

Источник

Grouped by the size of the batch each request actually ran in, the line is sharp. In the first run at 8, none of the 116 requests that ran alone or in pairs changed, while 311 of the 735 that ran in batches of seven did.

Источник

The effect on decisions was small. Across the three batched runs, 2 of 2,598 answers changed their top choice, and one of those turned a right answer into a wrong one.

Источник

We should have explained this the first time. "Automatable at 5% error" doesn't use the 90% threshold.

Источник