Claude Code's eval grader can't see a no‑break space

Description and use

AI grading tools misinterpret U+00A0 no‑break spaces, causing incorrect pass/fail decisions on whitespace‑sensitive code. Adding an explicit list of invisible characters to the answer corrects the issue.

Problem

AI grading tools misinterpret no‑break spaces as ordinary spaces, causing incorrect pass/fail decisions on coding tasks that require strict whitespace handling.

Outcome

Adding an explicit list of invisible characters to the answer text enabled AI graders to correctly interpret whitespace, raising accuracy from 0/6 to 6/6 for multiple models.

Workflow

• 1. Identify a coding question that involves strict whitespace handling.
• 2. Provide two candidate answers, one correct (Answer A) and one incorrect (Answer B).
• 3. Run the answers through multiple AI grading tools and record their decisions.
• 4. Observe that graders misinterpret no‑break spaces and fail to detect hidden characters.
• 5. Add a short note to the answer that enumerates the invisible characters with line/column and Unicode code point.
• 6. Re‑run the grading tools; all now correctly evaluate the answers.
• 7. Submit the fix as pull requests to the relevant open‑source projects.

Evidence

The task says to refuse it. A refuses it, so its test says False.

Source

The graders got 0 of 6 right with Haiku 5.5 and 1 of 6 with Sonnet 5.5.

Source

The fix is deliberately small. Alongside the answer, a short note lists any unusual spaces and invisible characters, with their exact position.

Source

All graders correctly evaluated both answers.

Source