My AI‑vs‑human commit comparison had a control group of one day

Описание и применение

Сравнение AI‑генерируемых коммитов с обычными показало, что AI‑коммиты чаще содержат "fix"/"revert" в теме, но это связано с привычками именования и большим размером коммитов. После корректировки разница исчезает.

Задача

Assess whether AI‑generated commits differ from non‑AI commits in terms of fix/revert subject frequency, but the comparison is confounded by a single‑day control group and commit size.

Результат

AI commits initially appear to have a higher fix/revert subject rate, but this is largely due to naming conventions and larger commit size. After adjusting for these factors, the difference disappears.

Рабочий процесс

• 1. Identify AI commits by Co‑Authored‑By: Claude trailer. 2. Count commits whose subject starts with "fix" or "revert". 3. Compare percentages between AI and non‑AI commits. 4. Expand subject match to include any occurrence of fix/bug/revert. 5. Examine commit size (files touched) and overlap within 72 h. 6. Re‑evaluate metrics after normalizing for commit size and subject wording.

Доказательства

AI commit = the message has a Co‑Authored‑By: Claude trailer (245 commits). Everything else is "no trailer" (65 commits).

Источник

subject starts with fix/revert: AI commits 38% vs. no‑trailer 17%

Источник

57 of the 65 landed on a single day, June 5.

Источник

AI commits touch more files: 3.1 on average, against 1.7.

Источник