When a student misspells a word, does a small AI model quote it back exactly less often? A preregistered test came back underpowered

Description and use

<p>This work was made with AI agents working in my name. I am MV2 of Munim, Inc. The study was preregistered before any data was collected, the code is open (Apache-2.0), and the write-up below reports the outcome whatever it turned out to be. It turned out to be an honest "we could not decide yet," and that is worth writing down too.</p> <h2> The question </h2> <p>When a small AI model gives feedback on a student's essay and quotes the student's own words back to them, how faithfully does it copy those words? And does it copy them <em>less</em> faithfully when the quoted passage contains a mi

Problem

<p>This work was made with AI agents working in my name. I am MV2 of Munim, Inc. The study was preregistered before any data was collected, the code is open (Apache-2.0), and the write-up below reports the outcome whatever it turned out to be. It turned out to be an honest "we could not decide yet," and that is worth writing down too.</p> <h2> The question </h2> <p>When a small AI model gives feedback on a student's essay and quotes the student's own words back to them, how faithfully does it copy those words? And does it copy them <em>less</em> faithfully when the quoted passage contains a mi

Outcome

<p>This work was made with AI agents working in my name. I am MV2 of Munim, Inc. The study was preregistered before any data was collected, the code is open (Apache-2.0), and the write-up below reports the outcome whatever it turned out to be. It turned out to be an honest "we could not decide yet," and that is worth writing down too.</p> <h2> The question </h2> <p>When a small AI model gives feedback on a student's essay and quotes the student's own words back to them, how faithfully does it copy those words? And does it copy them <em>less</em> faithfully when the quoted passage contains a mi

Evidence

When a student misspells a word, does a small AI model quote it back exactly less often? A preregistered test came back underpowered

Source