My benchmark scored GPT-5.4 mini 1.00 by silently skipping the 3 questions it failed

Description and use

<p><em>This is a submission for the <a href="https://dev.to/challenges/kaggle-2026-09-23">Kaggle Benchmarking Challenge</a></em></p> <p>GPT-5.4 mini got a perfect 1.00 on my benchmark. Then, in a separate repeats notebook, I asked it the same three questions five times each with the cache off, and it failed all 15. The model didn't change. My benchmark had four bugs. Two made the leaderboard lie in both directions, one made a broken run look like a bad model, and one was my own checker misreading good answers.</p> <p>Spoiler: once the bugs were gone, one model refused a polite, allowed request

Problem

<p><em>This is a submission for the <a href="https://dev.to/challenges/kaggle-2026-09-23">Kaggle Benchmarking Challenge</a></em></p> <p>GPT-5.4 mini got a perfect 1.00 on my benchmark. Then, in a separate repeats notebook, I asked it the same three questions five times each with the cache off, and it failed all 15. The model didn't change. My benchmark had four bugs. Two made the leaderboard lie in both directions, one made a broken run look like a bad model, and one was my own checker misreading good answers.</p> <p>Spoiler: once the bugs were gone, one model refused a polite, allowed request

Outcome

<p><em>This is a submission for the <a href="https://dev.to/challenges/kaggle-2026-09-23">Kaggle Benchmarking Challenge</a></em></p> <p>GPT-5.4 mini got a perfect 1.00 on my benchmark. Then, in a separate repeats notebook, I asked it the same three questions five times each with the cache off, and it failed all 15. The model didn't change. My benchmark had four bugs. Two made the leaderboard lie in both directions, one made a broken run look like a bad model, and one was my own checker misreading good answers.</p> <p>Spoiler: once the bugs were gone, one model refused a polite, allowed request

Evidence

My benchmark scored GPT-5.4 mini 1.00 by silently skipping the 3 questions it failed

Source