How we use Jev to answer quiz questions (a 90-question benchmark)

Описание и применение

В статье описывается интеграция модели Jev в расширение QuizPilot для решения вопросов викторин. Jev достиг 85–86 из 90 правильных ответов за 0,17–0,44 с, значительно быстрее и с более точными метриками уверенности, чем обычные LLM.

Задача

Integrating a decision model (Jev) into a quiz‑answering Chrome extension to improve accuracy and speed over general LLMs.

Результат

Jev answered 85–86 of 90 questions correctly across two runs, outperforming a lightweight LLM (60/90) and matching a reasoning LLM (90/90) while being 15–50× faster.

Рабочий процесс

• 1. Identify question type (single choice, true/false, multi‑select).
• 2. Convert to Jev format: single choice → categorical, true/false → yes/no, multi‑select → multiple yes/no questions.
• 3. Send a single request containing all questions to Jev via OpenRouter.
• 4. Receive choice probabilities and compute confidence (|2p‑1| for yes/no, lowest confidence for multi‑select).
• 5. If confidence < 0.6, flag for re‑check or fallback to lightweight LLM.
• 6. Aggregate results and present suggested answers in the extension UI.

Доказательства

Jev answered 85–86 of 90 questions correctly across two runs.

Источник

one Jev request answered 60 questions in 0.17–0.44 s.

Источник

All nine mistakes were single-choice questions whose answer has to be computed, not recognised.

Источник

Re-checking everything below 0.6 would have touched 14 of 180 answers (8%).

Источник