LLMs Pass the Data-Science Quiz, Then Give Different Advice: A Kaggle Benchmark of 36 Measured Judgment Calls

说明与用途

LLM 在多选数据科学判断测试中表现良好，但在自由文本回答中仅有56–81%的正确率，常常误解原因。

问题

Data‑science practitioners often rely on rule‑of‑thumb judgments that may conflict with empirical measurements, leading to suboptimal decisions in Kaggle competitions.

结果

All models achieved near‑perfect scores on the multiple‑choice task but lost 6–16 cases on the open‑ended task, revealing a gap between recognition and reasoning.

工作流

• 1. Define 12 topics with 3 numeric variants each to create 36 judgment cases.
• 2. Collect measured answers from Kaggle Playground S6E9 and S6E10.
• 3. Design two tasks: multiple‑choice recognition (ds‑judgment) and open‑ended generation (ds‑judgment‑open).
• 4. Run a suite of 11 LLMs via Kaggle Model Proxy.
• 5. Record correct/incorrect per case and analyze failure patterns.
• 6. Adjust benchmark scoring to return (correct,total) per case.
• 7. Interpret results and identify common failure themes.

证据

Correct out of 36: ... 35, 36, 34, 35, 36, 36, 35, 34, 34, 34, 36, 35

来源

Models correctly say "revert to 5-fold, not worth the compute", but explain it as "the +0.00012 is noise".

来源

The judge is a single model (Gemini 3.8 Flash), which could favour its own family's phrasing.

来源