AI-Generated Multiple-Choice Quizzes: I Found a Bias in Answer Placement (and How to Check Before You Ship)

Description and use

The author found a sequential bias in AI‑generated multiple‑choice questions and recommends three checks—AI solving, answer position analysis, and full field review—to reduce bias and prevent auto‑fill errors before publishing on Udemy.

Problem

AI-generated multiple‑choice questions can contain systematic biases that make them easy to guess and may lead to platform policy violations.

Outcome

The author reduced positional bias from near 100% to 32% and avoided platform policy issues by thoroughly reviewing all fields, leading to a cleaner, more reliable practice quiz.

Workflow

• Generate 300 questions with AI (4 options each)
• Feed questions without answers to a different AI to solve and flag suspicious ones
• Remove flagged questions
• Count correct answer positions to detect sequential patterns
• Scramble option order where bias >25% and enable platform randomization
• Manually review all fields after saving, including auto‑filled ones
• Publish 120 questions on Udemy
• Set up weekly alerts to compare published page text with approved version

Evidence

I fed the questions (without answers or explanations) to a different AI than the one that generated them. The result: 300 out of 300 questions were answered correctly by the second AI.

Source

What I found was a clear pattern: the correct answer often cycled sequentially, like 1 → 2 → 3 → 4 → 1... For adjacent questions, the correct answer's position "advanced by one" in 58% of cases.

Source

I had checked the character count for the description, but I hadn't looked at the adjacent field that I hadn't explicitly typed into.

Source