My quote filter hid three quarters of the quotes: checking AI feedback ranked 'more specific'

说明与用途

一个原本用于去除重写提示的引文过滤器在SpecEval数据集中错误地删除了大量真实学生引文，导致可信度指标失真。通过将引用短语视为合法引文来修订过滤器，保留的引文数量增加了三倍，误引率下降，证实更高排名的反馈更可信，但仍有1/40的顶级引文会更改学生原话。

问题

A quote‑filter designed to remove suggested rewrites was unintentionally discarding genuine student quotes in the SpecEval dataset, leading to an under‑representation of quoted passages and skewed fidelity metrics.

结果

The revised filter increased retained quotes threefold and reduced the misquote rate, revealing that "more specific" feedback is also more faithful, though still 1 in 40 top‑ranked quotes altered student words.

工作流

• 1. Identify quoted passages in SpecEval feedback pairs.
• 2. Apply initial filter that removes quotes preceded by rewrite cues.
• 3. Count retained quotes and measure misquote rate (quotes that change student words).
• 4. Observe low retention and high misquote rate.
• 5. Revise filter to treat citation phrases as valid quotes (follow‑up 5b).
• 6. Re‑count retained quotes and recalculate misquote rates.
• 7. Compare misquote rates between top‑ranked and bottom‑ranked feedback.
• 8. Document findings and update code repository.

证据

Only 519 quotes of four or more words passed my filters, out of 14,413 quoted passages.

来源

I wrote a second plan, follow‑up 5b, before counting again. It held up: 1,546 matched quotes sit behind only a citation phrase, three times as many as the 519 audit 5 kept.

来源

The top‑ranked feedback looked more faithful (3.4% against 12.2%), but that rested on 16 changed quotes, and the floor is there so I don’t read anything into numbers like that.

来源