SQuAD's answer normalizer can mark a wrong answer correct: three verified hazards

说明与用途

SQuAD 的 normalize_answer 可能导致错误的评估结果，本文记录了三种具体风险并提供了可复现的数据集。

问题

SQuAD’s normalize_answer function can cause scoring anomalies, marking incorrect answers as correct or missing correct ones.

结果

Three concrete hazards were identified: empty‑collapse false positives, disagreement between squad and squad_v2 on the same data, and punctuation gluing articles, all of which can mislead evaluation metrics.

工作流

• Identify the normalization function used in the evaluation pipeline.
• Create a labeled, execution‑verified corpus of problematic cases.
• Run the corpus against the SQuAD scorer to observe scoring anomalies.
• Use the reproduced tests to validate fixes or alternative metrics.
• Share the findings and code with the community.

证据

The exact‑match and F1 scores in most question‑answering and many open‑ended LLM evals run every answer through one tiny, famous function first: SQuAD’s normalize_answer.

来源

A gold answer that is a bare article (a, an, the) or punctuation‑only (…) normalizes to the empty string. The v1.1 scorer builds its gold list without filtering and compares by equality, so any prediction that also normalizes to empty — an article, punctuation, whitespace, even an empty refusal — scores exact‑match 1.0.

来源

squad_v2 does filter golds that normalize to empty. So with a real co‑alias — gold ["the", "red car"], prediction "a" — squad scores 1.0 while squad_v2 scores 0.

来源

remove_punc runs before remove_articles and deletes punctuation without inserting a space. So a missing space keeps the article attached… A model that writes end.the or the-car scores 0 against a gold the normalizer would otherwise have made equal — a false negative that depends only on spacing.

来源

python -m unittest -v test_squad_normalize_hazards.py

来源