My benchmark scored GPT-5.4 mini 1.00 by silently skipping the 3 questions it failed

Описание и применение

<p><em>This is a submission for the <a href="https://dev.to/challenges/kaggle-2026-09-23">Kaggle Benchmarking Challenge</a></em></p> <p>GPT-5.4 mini got a perfect 1.00 on my benchmark. Then, in a separate repeats notebook, I asked it the same three questions five times each with the cache off, and it failed all 15. The model didn't change. My benchmark had four bugs. Two made the leaderboard lie in both directions, one made a broken run look like a bad model, and one was my own checker misreading good answers.</p> <p>Spoiler: once the bugs were gone, one model refused a polite, allowed request

Задача

<p><em>This is a submission for the <a href="https://dev.to/challenges/kaggle-2026-09-23">Kaggle Benchmarking Challenge</a></em></p> <p>GPT-5.4 mini got a perfect 1.00 on my benchmark. Then, in a separate repeats notebook, I asked it the same three questions five times each with the cache off, and it failed all 15. The model didn't change. My benchmark had four bugs. Two made the leaderboard lie in both directions, one made a broken run look like a bad model, and one was my own checker misreading good answers.</p> <p>Spoiler: once the bugs were gone, one model refused a polite, allowed request

Результат

<p><em>This is a submission for the <a href="https://dev.to/challenges/kaggle-2026-09-23">Kaggle Benchmarking Challenge</a></em></p> <p>GPT-5.4 mini got a perfect 1.00 on my benchmark. Then, in a separate repeats notebook, I asked it the same three questions five times each with the cache off, and it failed all 15. The model didn't change. My benchmark had four bugs. Two made the leaderboard lie in both directions, one made a broken run look like a bad model, and one was my own checker misreading good answers.</p> <p>Spoiler: once the bugs were gone, one model refused a polite, allowed request

Доказательства

My benchmark scored GPT-5.4 mini 1.00 by silently skipping the 3 questions it failed

Источник