When Simpler AI Decisions Worked Better: What I Changed in My Jev-Powered Game Judge

Description and use

Simplifying the decision logic in the Sideways game, which used Jev, led to more accurate and flexible responses, improving player interaction.

Problem

The AI‑powered Game Master in the Sideways puzzle game produced overly rigid responses, causing player queries to be rejected or answered poorly despite the model understanding the intent.

Outcome

By simplifying the decision logic and reducing reliance on strict probability thresholds, the Game Master returned more accurate and helpful answers, improving player engagement.

Workflow

• 1. Build a semantic judgment model (Jev) that outputs probabilities for well‑formedness, relevance, evidence, etc.
• 2. Translate these probabilities into deterministic outcomes using TypeScript rules with 0.8 thresholds.
• 3. Observe that many player queries are rejected or answered poorly despite high model confidence.
• 4. Identify that hard gates and conservative thresholds cause loss of useful information.
• 5. Refactor the decision logic: lower thresholds, remove unnecessary hard gates, and simplify the mapping from probabilities to outcomes.
• 6. Deploy Version 3 and test with real player queries, noting improved response quality.

Evidence

Making those semantic IF statements more sophisticated actually made the game worse.

Source

The word mechines is obviously a typo. But the intended meaning is still easy for a human to recover.

Source

0.7999 → rejected 0.8000 → accepted

Source