Can AI tell whether an optimum is unique? A small Kaggle benchmark with exact answers

Описание и применение

Модель Gemini 3.7 Flash смогла определить уникальность оптимума и посчитать их количество в 70–75 % случаев, а GPT‑6.1‑sol точно посчитала 10 сложных задач.

Задача

Determine if language models can identify whether an optimization problem’s optimum is unique and, if not, count the number of optimal solutions.

Результат

Gemini 3.7 Flash achieved 0.90 on the leaderboard (26/29 correct) while gpt-oss-120b scored 0.55 (16/29). In a full run on 87 instances, Gemini correctly counted optima 70% of the time with a plain prompt and 75% with an explicit prompt. Counting accuracy decreased with more optima, and wrong counts were usually underestimates. GPT-6.1-sol matched all 10 hard cases exactly.

Рабочий процесс

• 1. Generate 87 small optimization instances (subset selection, QUBO, assignment, knapsack, toy portfolio).
• 2. Enumerate each instance exhaustively to obtain ground truth (optimal value and exact number of optima).
• 3. Prompt AI models to answer in free text ending with a JSON line containing solution, objective, unique flag, and number of optima.
• 4. Recompute the objective of the returned solution to verify optimality.
• 5. Compare the model’s uniqueness claim and count against ground truth.
• 6. Score each instance: 1 if all three checks pass, otherwise 0.
• 7. Aggregate scores over the 29-instance leaderboard board to compute leaderboard value.
• 8. Repeat with different prompts and models to analyze performance variations.

Доказательства

Gemini 3.7 Flash scored 0.90 (26 of 29) and gpt-oss-120b scored 0.55 (16 of 29).

Источник

In the full run on 87 instances, Gemini 3.7 Flash got the exact number of optima right in 70% of answers with the plain prompt and 75% with the explicit prompt.

Источник

When the count is wrong, it is usually too low. 43 of the 48 wrong counts were below the truth.

Источник