It’s Not the Model, It’s You

说明与用途

模型选择对输出差异的影响不到1%，主要驱动因素是个人习惯、团队文化和使用量。

问题

Engineering teams often assume that switching to a newer frontier AI model will significantly improve coding output, but evidence shows the model choice has minimal impact compared to other factors.

结果

Model selection accounts for <1% of output variance; 9% from usage volume, 12% from team/culture, >75% from individual engineer factors.

工作流

• Collect telemetry on AI tool usage across teams
• Analyze model switch patterns and output metrics
• Quantify contribution of model, usage volume, team, and individual factors
• Interpret results to guide leadership focus

证据

We analyze and extract industry insights from our massive collection of software engineering workflow data, which tracks AI coding tool and agent telemetry across 300,000 developers from 1,300 companies.

来源

There wasn’t. Comparing the engineers who moved the most onto Grok with colleagues at their own companies who barely touched it (more than 12,000 engineers at 222 companies), there was essentially no difference in output.

来源

Less than 1%: The tool itself (model and harness together). About 9%: The volume of AI that engineers use. About 12%: Their company, team, and culture. More than 75%: The individual engineer: their role, their track record, their habits, and everything else that separates today’s great AI engineers from those still figuring it out.

来源