I built a Jev-style decision model for Japanese in two days. Claude Code A/B-tested my architecture and killed it.

说明与用途

构建了一个日本语的 Jev 风格决策模型，优于多语言基线，消除了位置偏差，并在路由、紧急度评分和流失检测任务上实现了高准确率。

问题

Japanese business emails require multi‑class routing, urgency scoring, and churn detection, but existing multilingual System One models (e.g., laya‑multilingual) exhibit positional bias and poor ordinal performance, making them unreliable for production.

结果

The custom Japanese System One model outperforms the multilingual baseline on all primitives, eliminates positional bias, and demonstrates that careful data design and architecture choice are critical for intent‑based classification.

工作流

• 1. Generate 300 Japanese business emails with a local LLM (Qwen3 30B‑A3B) using label‑conditioned prompts.
• 2. Define three primitives: choice (4‑way routing), score (3‑level urgency), bool (churn hint).
• 3. Benchmark laya‑multilingual and baselines on the 300 items.
• 4. Identify positional bias in score predictions via five ordering conditions.
• 5. Build sokudan‑ja‑310m: fine‑tune ModernBERT‑Ja with one <mask> per option and a cumulative‑link score head.
• 6. Evaluate on the same 300 items with three random seeds.
• 7. Diagnose bool performance: replace input text with empty string to reveal prior bias.
• 8. Redesign data with 24 attributes split into surface, explicit attitude, and implicit intent; hold out five attributes for validation.
• 9. Retrain and evaluate; observe improved bool AUROC and reduced bias.
• 10. Publish model, code, and benchmark scripts on Hugging Face and GitHub.

证据

sokudan‑ja‑310m achieved 0.847 ± 0.009 choice accuracy, 0.763 ± 0.088 score accuracy, and 0.788 ± 0.010 bool accuracy on 300 items.

来源

The lowest level is never predicted. My first guesses were "weak on Japanese negation" or "ordinal training is broken".

来源