AI-Generated Tests: How Many Actually Test Different Behaviors?

Description and use

AI‑generated tests often duplicate each other without testing distinct behaviors. vitest‑dedup 0.5.0 distinguishes identical, near‑duplicate, and truly distinct tests, improving coverage quality.

Problem

Automated test generation can produce many passing tests that appear to increase coverage, but many of those tests are duplicates or near‑duplicates that do not test distinct behaviors.

Outcome

vitest‑dedup 0.5.0 correctly distinguishes between identical tests, near‑duplicates, and tests that exercise different behaviors, whereas a naive token‑based detector misclassifies many cases.

Workflow

• Run coding agent to generate tests
• Classify suspicious pairs manually
• Run three detectors: naive token‑based, vitest‑dedup 0.4.0, vitest‑dedup 0.5.0
• Compare results and analyze verdicts

Evidence

A coding agent can write a hundred passing tests before you finish your coffee. The test count goes up, the coverage report turns green, and the pull request looks thorough.

Source

vitest-dedup encodes this split in its output. Every finding carries a verdict: redundant, consolidatable, or a review prompt.

Source

The naive detector reports them as exact duplicates. The bodies are the same text, but clock is a different binding in each one, and that binding is the test.

Source

vitest-dedup 0.5.0 reports neither pair. Captured variables participate in the comparison at the place they are used, and they keep track of which enclosing variable they came from.

Source

0.5.0 models local classes: fields, methods, constructors, private names and what each instance captures. Consistent renaming stays comparable, so the verdict becomes a proven duplicate-test.

Source