It’s Not the Model, It’s You

Description and use

Model choice explains less than 1% of output variance; individual engineer habits, team culture, and usage volume drive most productivity differences.

Problem

Engineering teams often assume that switching to a newer frontier AI model will significantly improve coding output, but evidence shows the model choice has minimal impact compared to other factors.

Outcome

Model selection accounts for <1% of output variance; 9% from usage volume, 12% from team/culture, >75% from individual engineer factors.

Workflow

• Collect telemetry on AI tool usage across teams
• Analyze model switch patterns and output metrics
• Quantify contribution of model, usage volume, team, and individual factors
• Interpret results to guide leadership focus

Evidence

We analyze and extract industry insights from our massive collection of software engineering workflow data, which tracks AI coding tool and agent telemetry across 300,000 developers from 1,300 companies.

Source

There wasn’t. Comparing the engineers who moved the most onto Grok with colleagues at their own companies who barely touched it (more than 12,000 engineers at 222 companies), there was essentially no difference in output.

Source

Less than 1%: The tool itself (model and harness together). About 9%: The volume of AI that engineers use. About 12%: Their company, team, and culture. More than 75%: The individual engineer: their role, their track record, their habits, and everything else that separates today’s great AI engineers from those still figuring it out.

Source