My quote filter hid three quarters of the quotes: checking AI feedback ranked 'more specific'

Description and use

A quote‑filter tuned to remove rewrite cues mistakenly discarded genuine student quotes in the SpecEval dataset, skewing fidelity metrics. Revising the filter to treat citation phrases as valid quotes tripled retained quotes and lowered misquote rates, confirming that higher‑ranked feedback is more faithful, though 1 in 40 top quotes still alters student words.

Problem

A quote‑filter designed to remove suggested rewrites was unintentionally discarding genuine student quotes in the SpecEval dataset, leading to an under‑representation of quoted passages and skewed fidelity metrics.

Outcome

The revised filter increased retained quotes threefold and reduced the misquote rate, revealing that "more specific" feedback is also more faithful, though still 1 in 40 top‑ranked quotes altered student words.

Workflow

• 1. Identify quoted passages in SpecEval feedback pairs.
• 2. Apply initial filter that removes quotes preceded by rewrite cues.
• 3. Count retained quotes and measure misquote rate (quotes that change student words).
• 4. Observe low retention and high misquote rate.
• 5. Revise filter to treat citation phrases as valid quotes (follow‑up 5b).
• 6. Re‑count retained quotes and recalculate misquote rates.
• 7. Compare misquote rates between top‑ranked and bottom‑ranked feedback.
• 8. Document findings and update code repository.

Evidence

Only 519 quotes of four or more words passed my filters, out of 14,413 quoted passages.

Source

I wrote a second plan, follow‑up 5b, before counting again. It held up: 1,546 matched quotes sit behind only a citation phrase, three times as many as the 519 audit 5 kept.

Source

The top‑ranked feedback looked more faithful (3.4% against 12.2%), but that rested on 16 changed quotes, and the floor is there so I don’t read anything into numbers like that.

Source