It’s Not the Model, It’s You

Описание и применение

Модель не решает большую часть проблемы производительности инженеров; ключевые факторы – это индивидуальные привычки, культура команды и объем использования AI.

Задача

Engineering teams often assume that switching to a newer frontier AI model will significantly improve coding output, but evidence shows the model choice has minimal impact compared to other factors.

Результат

Model selection accounts for <1% of output variance; 9% from usage volume, 12% from team/culture, >75% from individual engineer factors.

Рабочий процесс

• Collect telemetry on AI tool usage across teams
• Analyze model switch patterns and output metrics
• Quantify contribution of model, usage volume, team, and individual factors
• Interpret results to guide leadership focus

Доказательства

We analyze and extract industry insights from our massive collection of software engineering workflow data, which tracks AI coding tool and agent telemetry across 300,000 developers from 1,300 companies.

Источник

There wasn’t. Comparing the engineers who moved the most onto Grok with colleagues at their own companies who barely touched it (more than 12,000 engineers at 222 companies), there was essentially no difference in output.

Источник

Less than 1%: The tool itself (model and harness together). About 9%: The volume of AI that engineers use. About 12%: Their company, team, and culture. More than 75%: The individual engineer: their role, their track record, their habits, and everything else that separates today’s great AI engineers from those still figuring it out.

Источник