Decision models are English-first. I measured them in Japanese, then built one.

说明与用途

创建了日语决策模型sokudan，取得高选择准确率和低评分RPS。英文模型存在首选项偏差。跨语言迁移保持选择性能，但不适用于是/否问题。发布了日语typed‑decisions基准，回译匹配率0.946。

问题

Japanese users lack decision models that handle typed questions without output tokens, and existing English models exhibit a position prior that hurts ordinal questions.

结果

sokudan delivers high choice accuracy and low score RPS on Japanese messages, outperforming English models on ordinal questions. Position prior was identified and mitigated in Laya and other models. Japanese‑only training preserves choice performance in English but not yes/no.

工作流

• 1. Generate 300 synthetic Japanese support messages (bench_ja).
• 2. Run laya‑multilingual on bench_ja to identify position prior issues.
• 3. Report findings via GitHub issues and pull requests to add presentation checks.
• 4. Build sokudan: fine‑tune modernbert‑ja‑310m on synthetic data, encode state and question jointly, add cumulative‑link head for score.
• 5. Evaluate sokudan on bench_ja and compare with Lev, laya‑multilingual, and majority baseline.
• 6. Test sokudan on English bench to assess cross‑language transfer.
• 7. Translate LocalLLaMA typed‑decisions into Japanese, back‑translate, and filter by match flag.
• 8. Publish sokudan weights on Hugging Face and provide pip install instructions.
• 9. Offer a server exposing /v1/systemone API for local use.

证据

laya-multilingual never picked the first level of an ordinal question in Japanese: 0 of 300. Same in English: 0 of 290.

来源

I built sokudan, an open 310M Japanese decision model. On 300 Japanese business messages: choice accuracy 0.880, score RPS 0.075, bool AUROC 0.844.

来源

Trained only on Japanese, its choice transfers to English almost intact; its yes / no does not.

来源

Machine translation needs checking, so every case was back‑translated and judged against the source by a second model. Overall match: 0.946.

来源