Decision models are English-first. I measured them in Japanese, then built one.

Описание и применение

Создан японский модельный решатель sokudan, который демонстрирует высокую точность выбора и низкое отклонение оценки. Выявлен и устранен предвзятый выбор первой опции в английских моделях. Перенос результатов на английский язык ограничен для вопросов «да/нет».

Задача

Japanese users lack decision models that handle typed questions without output tokens, and existing English models exhibit a position prior that hurts ordinal questions.

Результат

sokudan delivers high choice accuracy and low score RPS on Japanese messages, outperforming English models on ordinal questions. Position prior was identified and mitigated in Laya and other models. Japanese‑only training preserves choice performance in English but not yes/no.

Рабочий процесс

• 1. Generate 300 synthetic Japanese support messages (bench_ja).
• 2. Run laya‑multilingual on bench_ja to identify position prior issues.
• 3. Report findings via GitHub issues and pull requests to add presentation checks.
• 4. Build sokudan: fine‑tune modernbert‑ja‑310m on synthetic data, encode state and question jointly, add cumulative‑link head for score.
• 5. Evaluate sokudan on bench_ja and compare with Lev, laya‑multilingual, and majority baseline.
• 6. Test sokudan on English bench to assess cross‑language transfer.
• 7. Translate LocalLLaMA typed‑decisions into Japanese, back‑translate, and filter by match flag.
• 8. Publish sokudan weights on Hugging Face and provide pip install instructions.
• 9. Offer a server exposing /v1/systemone API for local use.

Доказательства

laya-multilingual never picked the first level of an ordinal question in Japanese: 0 of 300. Same in English: 0 of 290.

Источник

I built sokudan, an open 310M Japanese decision model. On 300 Japanese business messages: choice accuracy 0.880, score RPS 0.075, bool AUROC 0.844.

Источник

Trained only on Japanese, its choice transfers to English almost intact; its yes / no does not.

Источник

Machine translation needs checking, so every case was back‑translated and judged against the source by a second model. Overall match: 0.946.

Источник