Jev vs Local Models: Where Japanese Intent Routing Stands After 1,000 Utterances

Описание и применение

Jev показал 97,4 % точности в классификации японских коротких фраз, превосходя локальные модели, особенно в продолжениях и одиночных запросах. Qwen3.6 лучше справился с переключением задач.

Задача

Japanese short utterance intent routing accuracy varies across models; need to evaluate which model best classifies utterances for a voice assistant.

Результат

Jev outperformed local models overall, especially on continuations (99.6%) and single utterances (97.1%). Qwen3.6 excelled on task switches (98.8%). Local models lagged in small talk and ambiguous utterances.

Рабочий процесс

• Collect 1,000 Japanese utterances, replacing 43 overlapping cases; 66 ambiguous cases counted separately.
• Create correct answer labels via two independent assigners; 956/957 consistency.
• Run each model on each utterance with a single‑line context for continuations; temperature 0; JSON output format.
• Record model outputs, confidence, response time, and correctness.
• Aggregate results per model and per utterance type.
• Perform McNemar test on Jev vs Qwen3.6.
• Publish scripts and case data for reproducibility.

Доказательства

Jev alone answered correctly in 55 cases, Qwen alone answered correctly in 10 cases, and both failed in 14 cases. The McNemar test (exact test) resulted in p ≈ 1.2×10⁻⁸.

Источник

In the final 934 cases, categorized by type, there was a significant difference in the continuation judgment. Jev achieved 99.6% out of 233 cases, Qwen3.6 achieved 86.3%, and both qwen3:8b and qwen3.5:4b achieved 70.4%.

Источник

The utterances and correct answers for reproduction are in scripts/jev-route-k1000-cases.json, and the measurement script is in scripts/eval-jev.ts.

Источник