Jev vs Local Models: Where Japanese Intent Routing Stands After 1,000 Utterances

Description and use

Jev achieved 97.4% accuracy on 934 Japanese utterances, outperforming local models, especially on continuations and single utterances. Qwen3.6 excelled at task switching.

Problem

Japanese short utterance intent routing accuracy varies across models; need to evaluate which model best classifies utterances for a voice assistant.

Outcome

Jev outperformed local models overall, especially on continuations (99.6%) and single utterances (97.1%). Qwen3.6 excelled on task switches (98.8%). Local models lagged in small talk and ambiguous utterances.

Workflow

• Collect 1,000 Japanese utterances, replacing 43 overlapping cases; 66 ambiguous cases counted separately.
• Create correct answer labels via two independent assigners; 956/957 consistency.
• Run each model on each utterance with a single‑line context for continuations; temperature 0; JSON output format.
• Record model outputs, confidence, response time, and correctness.
• Aggregate results per model and per utterance type.
• Perform McNemar test on Jev vs Qwen3.6.
• Publish scripts and case data for reproducibility.

Evidence

Jev alone answered correctly in 55 cases, Qwen alone answered correctly in 10 cases, and both failed in 14 cases. The McNemar test (exact test) resulted in p ≈ 1.2×10⁻⁸.

Source

In the final 934 cases, categorized by type, there was a significant difference in the continuation judgment. Jev achieved 99.6% out of 233 cases, Qwen3.6 achieved 86.3%, and both qwen3:8b and qwen3.5:4b achieved 70.4%.

Source

The utterances and correct answers for reproduction are in scripts/jev-route-k1000-cases.json, and the measurement script is in scripts/eval-jev.ts.

Source