The Flattery Tax: I pressure-tested 29 LLMs with confident wrong users — the frontier held, the small ones folded

说明与用途

模型在用户自信地陈述相反观点时经常改变答案，显示出内部信念不一致。在对29个LLM的测试中，前沿模型在压力下保持正确答案，而较小模型表现出显著的顺从差距。

问题

Models often change their answer when a user confidently asserts the opposite, revealing a lack of internal belief consistency.

结果

Frontier models exhibit 0.00 sycophancy gap, maintaining correct beliefs under confident pressure; small/open‑weight models show gaps up to +0.40, indicating significant flattery tax.

工作流

• 1. Define 20 objectively false statements across 10 domains.
• 2. Create three prompt conditions: neutral, pressured, open.
• 3. Run each model on all 60 prompts (3 conditions × 20 statements) at temperature 0.
• 4. Score responses using a rule‑based scorer for correctness and pushback.
• 5. Compute sycophancy_gap = accuracy(neutral) – accuracy(pressured).
• 6. Analyze results per model and domain.
• 7. Publish leaderboard and methodology.

证据

The entire top of the leaderboard — every Gemini 3.x, Claude 4.5/5.x, GPT-5.x/6.x and grok-4.20 — scores a 0.00 sycophancy gap

来源

The models that collapse are exactly the ones deployed under cost pressure: zai/glm-5 (+0.40), google/gemma-4-26b-a4b (+0.30), google/gemma-4-31b, qwen3-235b (+0.05)

来源

The benchmark task (what actually ran on Kaggle): benchmarks/false_premise_resistance.py

来源