Jev vs Claude for OpenPoke: Email Screening Speed and Cost

Описание и применение

Внедрение модели Jev в OpenPoke ускорило принятие решения о важности письма почти в 6 раз и сократило стоимость до 95‑раз меньшей по сравнению с Claude Sonnet, сохраняя при этом высокий уровень согласия.

Задача

Improve email triage speed and reduce cost in the OpenPoke assistant by replacing the default chat model with Jev for the screening decision.

Результат

Using Jev for the screening decision in OpenPoke reduced mean latency from 2,452 ms to 424 ms and cut input cost from $4.63 to $0.049 per 1,000 screens, while maintaining high agreement with Claude Sonnet. The injection gate achieved a 95.3 % trigger rate against hostile text, preventing low‑priority messages from surfacing.

Рабочий процесс

• 1. Fork OpenPoke and add Jev integration. 2. Replace the single boolean call with a Jev call that asks four questions: importance, security‑code content, bulk mail, and AI‑assistant instructions. 3. Route decisions: if Jev deems mail unimportant, skip the LLM; if important, proceed to summarisation; if uncertain, fall back to the original classifier. 4. Add Jev checks before tool execution and for search relevance. 5. Run evaluation harnesses on 12 synthetic emails, measuring latency and cost for both Claude Sonnet and Jev. 6. Perform a prompt‑injection test with hostile text to validate the injection gate. 7. Deploy the fork via the GitHub repo and enable the TYPESAFE_API_KEY for live operation.

Доказательства

Mean screening latency went from 2,452 ms with Claude Sonnet to 424 ms with Jev.

Источник

Estimated input cost went from $4.63 to $0.049 per 1,000 screens.

Источник

The injection gate fired in 95.3% of attack trials across 12 synthetic carriers.

Источник