Claude Code's eval grader can't see a no‑break space

Описание и применение

В оценочных инструментах ИИ не распознаются символы U+00A0, что приводит к неверным результатам проверки кода, требующего строгого управления пробелами. Добавление списка невидимых символов в ответ исправляет ошибку.

Задача

AI grading tools misinterpret no‑break spaces as ordinary spaces, causing incorrect pass/fail decisions on coding tasks that require strict whitespace handling.

Результат

Adding an explicit list of invisible characters to the answer text enabled AI graders to correctly interpret whitespace, raising accuracy from 0/6 to 6/6 for multiple models.

Рабочий процесс

• 1. Identify a coding question that involves strict whitespace handling.
• 2. Provide two candidate answers, one correct (Answer A) and one incorrect (Answer B).
• 3. Run the answers through multiple AI grading tools and record their decisions.
• 4. Observe that graders misinterpret no‑break spaces and fail to detect hidden characters.
• 5. Add a short note to the answer that enumerates the invisible characters with line/column and Unicode code point.
• 6. Re‑run the grading tools; all now correctly evaluate the answers.
• 7. Submit the fix as pull requests to the relevant open‑source projects.

Доказательства

The task says to refuse it. A refuses it, so its test says False.

Источник

The graders got 0 of 6 right with Haiku 5.5 and 1 of 6 with Sonnet 5.5.

Источник

The fix is deliberately small. Alongside the answer, a short note lists any unusual spaces and invisible characters, with their exact position.

Источник

All graders correctly evaluated both answers.

Источник