Your exact_match metric's normalization options can both invent matches and hide real ones

Описание и применение

Метрика exact_match из HuggingFace может выдавать неверные результаты из‑за своих опций нормализации, приводя к ложным срабатываниям и пропущенным совпадениям.

Задача

HuggingFace evaluate’s exact_match metric can produce misleading scores due to its normalization knobs, leading to false positives and false negatives.

Результат

The study demonstrates that exact_match’s normalization options can both inflate scores by erasing distinguishing content and deflate scores by ignoring non-ASCII characters, highlighting the need for careful configuration.

Рабочий процесс

• Identify the exact_match metric in the evaluation pipeline.
• Review the four normalization options and their default order (regexes, lowercase, punctuation, digits).
• Test the metric with representative answer pairs to detect false positives (e.g., numeric or symbolic answers) and false negatives (e.g., non-ASCII punctuation).
• Adjust or disable specific normalization knobs based on the evaluation context.
• Re-run the evaluation to confirm that scores now align with expected outcomes.
• Document the chosen settings and rationale for future reproducibility.

Доказательства

HuggingFace evaluate's exact_match is one of the most-used metrics in LLM and QA evals.

Источник

It ships with four normalization knobs -- regexes_to_ignore, ignore_case, ignore_punctuation, ignore_numbers -- and each quietly changes which answer pairs count as equal before the comparison.

Источник

When a normalization option strips away everything that distinguishes two answers, both collapse to the same residue and score 1.0.

Источник

Punctuation and digit stripping only touch ASCII. So two answers that look identical can score 0.0 because one carries a non-ASCII twin that survives.

Источник

The straight apostrophe, ASCII hyphen and ASCII 4 are removed; their typographic / non-Latin counterparts are kept, so the strings no longer match.

Источник