Jev vs Claude for OpenPoke: Email Screening Speed and Cost

Description and use

Integrating Jev into OpenPoke speeds up email triage by ~5.8× and cuts cost to ~1/95 of Claude Sonnet, while maintaining high agreement.

Problem

Improve email triage speed and reduce cost in the OpenPoke assistant by replacing the default chat model with Jev for the screening decision.

Outcome

Using Jev for the screening decision in OpenPoke reduced mean latency from 2,452 ms to 424 ms and cut input cost from $4.63 to $0.049 per 1,000 screens, while maintaining high agreement with Claude Sonnet. The injection gate achieved a 95.3 % trigger rate against hostile text, preventing low‑priority messages from surfacing.

Workflow

• 1. Fork OpenPoke and add Jev integration. 2. Replace the single boolean call with a Jev call that asks four questions: importance, security‑code content, bulk mail, and AI‑assistant instructions. 3. Route decisions: if Jev deems mail unimportant, skip the LLM; if important, proceed to summarisation; if uncertain, fall back to the original classifier. 4. Add Jev checks before tool execution and for search relevance. 5. Run evaluation harnesses on 12 synthetic emails, measuring latency and cost for both Claude Sonnet and Jev. 6. Perform a prompt‑injection test with hostile text to validate the injection gate. 7. Deploy the fork via the GitHub repo and enable the TYPESAFE_API_KEY for live operation.

Evidence

Mean screening latency went from 2,452 ms with Claude Sonnet to 424 ms with Jev.

Source

Estimated input cost went from $4.63 to $0.049 per 1,000 screens.

Source

The injection gate fired in 95.3% of attack trials across 12 synthetic carriers.

Source