My quote filter hid three quarters of the quotes: checking AI feedback ranked 'more specific'

Описание и применение

Фильтр цитат, изначально настроенный на удаление предложений‑переписок, в наборе SpecEval удалил большую часть настоящих цитат, что привело к искажению метрик достоверности. После пересмотра фильтра, учитывающего только цитирующие фразы, количество оставшихся цитат увеличилось в три раза, а процент искаженных цитат снизился, подтверждая, что более «конкретный» отклик также более достоверен, но всё ещё 1 из 40 цитат изменяет слова студента.

Задача

A quote‑filter designed to remove suggested rewrites was unintentionally discarding genuine student quotes in the SpecEval dataset, leading to an under‑representation of quoted passages and skewed fidelity metrics.

Результат

The revised filter increased retained quotes threefold and reduced the misquote rate, revealing that "more specific" feedback is also more faithful, though still 1 in 40 top‑ranked quotes altered student words.

Рабочий процесс

• 1. Identify quoted passages in SpecEval feedback pairs.
• 2. Apply initial filter that removes quotes preceded by rewrite cues.
• 3. Count retained quotes and measure misquote rate (quotes that change student words).
• 4. Observe low retention and high misquote rate.
• 5. Revise filter to treat citation phrases as valid quotes (follow‑up 5b).
• 6. Re‑count retained quotes and recalculate misquote rates.
• 7. Compare misquote rates between top‑ranked and bottom‑ranked feedback.
• 8. Document findings and update code repository.

Доказательства

Only 519 quotes of four or more words passed my filters, out of 14,413 quoted passages.

Источник

I wrote a second plan, follow‑up 5b, before counting again. It held up: 1,546 matched quotes sit behind only a citation phrase, three times as many as the 519 audit 5 kept.

Источник

The top‑ranked feedback looked more faithful (3.4% against 12.2%), but that rested on 16 changed quotes, and the floor is there so I don’t read anything into numbers like that.

Источник