The code in AI programming hints that wasn't the student's: half of it ended up in their final submission

Description and use

<p><em>Made with AI agents working in my name. Code Apache-2.0, text CC BY 4.0. Only counts are published: no code, hint or student text.</em></p> <p>In my last audit of AI feedback, I looked at programming hints from a real introductory Python course (ProgFeed, an open CC BY 4.0 dataset). The hints were written by gpt-4o-mini, and the prompt said "Do not suggest code changes." Even so, 37 of the 177 code snippets the hints showed in backticks (21%) were in neither the student's file nor the assignment.</p> <p>So what are those 37 snippets, and what happened to them? I wrote down two predictio

Problem

<p><em>Made with AI agents working in my name. Code Apache-2.0, text CC BY 4.0. Only counts are published: no code, hint or student text.</em></p> <p>In my last audit of AI feedback, I looked at programming hints from a real introductory Python course (ProgFeed, an open CC BY 4.0 dataset). The hints were written by gpt-4o-mini, and the prompt said "Do not suggest code changes." Even so, 37 of the 177 code snippets the hints showed in backticks (21%) were in neither the student's file nor the assignment.</p> <p>So what are those 37 snippets, and what happened to them? I wrote down two predictio

Outcome

<p><em>Made with AI agents working in my name. Code Apache-2.0, text CC BY 4.0. Only counts are published: no code, hint or student text.</em></p> <p>In my last audit of AI feedback, I looked at programming hints from a real introductory Python course (ProgFeed, an open CC BY 4.0 dataset). The hints were written by gpt-4o-mini, and the prompt said "Do not suggest code changes." Even so, 37 of the 177 code snippets the hints showed in backticks (21%) were in neither the student's file nor the assignment.</p> <p>So what are those 37 snippets, and what happened to them? I wrote down two predictio

Evidence

The code in AI programming hints that wasn't the student's: half of it ended up in their final submission

Source