Jev vs Local Models: Where Japanese Intent Routing Stands After 1,000 Utterances

说明与用途

Jev 在 934 条日语短语分类中达 97.4% 的准确率，优于本地模型，尤其在连续句和单句上表现突出。Qwen3.6 在任务切换上表现更佳。

问题

Japanese short utterance intent routing accuracy varies across models; need to evaluate which model best classifies utterances for a voice assistant.

结果

Jev outperformed local models overall, especially on continuations (99.6%) and single utterances (97.1%). Qwen3.6 excelled on task switches (98.8%). Local models lagged in small talk and ambiguous utterances.

工作流

• Collect 1,000 Japanese utterances, replacing 43 overlapping cases; 66 ambiguous cases counted separately.
• Create correct answer labels via two independent assigners; 956/957 consistency.
• Run each model on each utterance with a single‑line context for continuations; temperature 0; JSON output format.
• Record model outputs, confidence, response time, and correctness.
• Aggregate results per model and per utterance type.
• Perform McNemar test on Jev vs Qwen3.6.
• Publish scripts and case data for reproducibility.

证据

Jev alone answered correctly in 55 cases, Qwen alone answered correctly in 10 cases, and both failed in 14 cases. The McNemar test (exact test) resulted in p ≈ 1.2×10⁻⁸.

来源

In the final 934 cases, categorized by type, there was a significant difference in the continuation judgment. Jev achieved 99.6% out of 233 cases, Qwen3.6 achieved 86.3%, and both qwen3:8b and qwen3.5:4b achieved 70.4%.

来源

The utterances and correct answers for reproduction are in scripts/jev-route-k1000-cases.json, and the measurement script is in scripts/eval-jev.ts.

来源