Your exact_match metric's normalization options can both invent matches and hide real ones

Description and use

HuggingFace’s exact_match metric can produce misleading scores due to its normalization options, leading to false positives and false negatives.

Problem

HuggingFace evaluate’s exact_match metric can produce misleading scores due to its normalization knobs, leading to false positives and false negatives.

Outcome

The study demonstrates that exact_match’s normalization options can both inflate scores by erasing distinguishing content and deflate scores by ignoring non-ASCII characters, highlighting the need for careful configuration.

Workflow

• Identify the exact_match metric in the evaluation pipeline.
• Review the four normalization options and their default order (regexes, lowercase, punctuation, digits).
• Test the metric with representative answer pairs to detect false positives (e.g., numeric or symbolic answers) and false negatives (e.g., non-ASCII punctuation).
• Adjust or disable specific normalization knobs based on the evaluation context.
• Re-run the evaluation to confirm that scores now align with expected outcomes.
• Document the chosen settings and rationale for future reproducibility.

Evidence

HuggingFace evaluate's exact_match is one of the most-used metrics in LLM and QA evals.

Source

It ships with four normalization knobs -- regexes_to_ignore, ignore_case, ignore_punctuation, ignore_numbers -- and each quietly changes which answer pairs count as equal before the comparison.

Source

When a normalization option strips away everything that distinguishes two answers, both collapse to the same residue and score 1.0.

Source

Punctuation and digit stripping only touch ASCII. So two answers that look identical can score 0.0 because one carries a non-ASCII twin that survives.

Source

The straight apostrophe, ASCII hyphen and ASCII 4 are removed; their typographic / non-Latin counterparts are kept, so the strings no longer match.

Source