Can AI tell whether an optimum is unique? A small Kaggle benchmark with exact answers

Description and use

Gemini 3.7 Flash identified optimum uniqueness and counted optima correctly 70–75 % of the time, while GPT‑6.1‑sol exactly counted 10 hard instances.

Problem

Determine if language models can identify whether an optimization problem’s optimum is unique and, if not, count the number of optimal solutions.

Outcome

Gemini 3.7 Flash achieved 0.90 on the leaderboard (26/29 correct) while gpt-oss-120b scored 0.55 (16/29). In a full run on 87 instances, Gemini correctly counted optima 70% of the time with a plain prompt and 75% with an explicit prompt. Counting accuracy decreased with more optima, and wrong counts were usually underestimates. GPT-6.1-sol matched all 10 hard cases exactly.

Workflow

• 1. Generate 87 small optimization instances (subset selection, QUBO, assignment, knapsack, toy portfolio).
• 2. Enumerate each instance exhaustively to obtain ground truth (optimal value and exact number of optima).
• 3. Prompt AI models to answer in free text ending with a JSON line containing solution, objective, unique flag, and number of optima.
• 4. Recompute the objective of the returned solution to verify optimality.
• 5. Compare the model’s uniqueness claim and count against ground truth.
• 6. Score each instance: 1 if all three checks pass, otherwise 0.
• 7. Aggregate scores over the 29-instance leaderboard board to compute leaderboard value.
• 8. Repeat with different prompts and models to analyze performance variations.

Evidence

Gemini 3.7 Flash scored 0.90 (26 of 29) and gpt-oss-120b scored 0.55 (16 of 29).

Source

In the full run on 87 instances, Gemini 3.7 Flash got the exact number of optima right in 70% of answers with the plain prompt and 75% with the explicit prompt.

Source

When the count is wrong, it is usually too low. 43 of the 48 wrong counts were below the truth.

Source