My AI‑vs‑human commit comparison had a control group of one day

Description and use

AI‑generated commits initially appear to have a higher fix/revert subject rate, but this is largely due to naming conventions and larger commit size. After adjusting for these factors, the difference disappears.

Problem

Assess whether AI‑generated commits differ from non‑AI commits in terms of fix/revert subject frequency, but the comparison is confounded by a single‑day control group and commit size.

Outcome

AI commits initially appear to have a higher fix/revert subject rate, but this is largely due to naming conventions and larger commit size. After adjusting for these factors, the difference disappears.

Workflow

• 1. Identify AI commits by Co‑Authored‑By: Claude trailer. 2. Count commits whose subject starts with "fix" or "revert". 3. Compare percentages between AI and non‑AI commits. 4. Expand subject match to include any occurrence of fix/bug/revert. 5. Examine commit size (files touched) and overlap within 72 h. 6. Re‑evaluate metrics after normalizing for commit size and subject wording.

Evidence

AI commit = the message has a Co‑Authored‑By: Claude trailer (245 commits). Everything else is "no trailer" (65 commits).

Source

subject starts with fix/revert: AI commits 38% vs. no‑trailer 17%

Source

57 of the 65 landed on a single day, June 5.

Source

AI commits touch more files: 3.1 on average, against 1.7.

Source