I built a Jev-style decision model for Japanese in two days. Claude Code A/B-tested my architecture and killed it.

Description and use

A Japanese System One decision model was built that outperforms the multilingual baseline, removes positional bias, and achieves high accuracy on routing, urgency scoring, and churn detection.

Problem

Japanese business emails require multi‑class routing, urgency scoring, and churn detection, but existing multilingual System One models (e.g., laya‑multilingual) exhibit positional bias and poor ordinal performance, making them unreliable for production.

Outcome

The custom Japanese System One model outperforms the multilingual baseline on all primitives, eliminates positional bias, and demonstrates that careful data design and architecture choice are critical for intent‑based classification.

Workflow

• 1. Generate 300 Japanese business emails with a local LLM (Qwen3 30B‑A3B) using label‑conditioned prompts.
• 2. Define three primitives: choice (4‑way routing), score (3‑level urgency), bool (churn hint).
• 3. Benchmark laya‑multilingual and baselines on the 300 items.
• 4. Identify positional bias in score predictions via five ordering conditions.
• 5. Build sokudan‑ja‑310m: fine‑tune ModernBERT‑Ja with one <mask> per option and a cumulative‑link score head.
• 6. Evaluate on the same 300 items with three random seeds.
• 7. Diagnose bool performance: replace input text with empty string to reveal prior bias.
• 8. Redesign data with 24 attributes split into surface, explicit attitude, and implicit intent; hold out five attributes for validation.
• 9. Retrain and evaluate; observe improved bool AUROC and reduced bias.
• 10. Publish model, code, and benchmark scripts on Hugging Face and GitHub.

Evidence

sokudan‑ja‑310m achieved 0.847 ± 0.009 choice accuracy, 0.763 ± 0.088 score accuracy, and 0.788 ± 0.010 bool accuracy on 300 items.

Source

The lowest level is never predicted. My first guesses were "weak on Japanese negation" or "ordinal training is broken".

Source