I let a small local model write the refactor tests I was too lazy to write

Description and use

DiffWitness uses a local Gemma 4 model to detect silent regressions after refactoring, automatically generating failing regression tests.

Problem

Refactor changes can introduce silent bugs that existing tests miss, especially edge cases like leading zeros in order IDs.

Outcome

DiffWitness identified a silent bug caused by the refactor, automatically generated a regression test, and demonstrated that a small local LLM can effectively find edge‑case regressions.

Workflow

• Load old and new function versions via TypeScript compiler API
• Instrument branches with counters
• Prompt Gemma 4 e4b to propose up to eight input values
• Validate and filter proposed inputs against type constraints
• Execute both versions in isolated VM contexts and compare results
• Shrink any detected differences to minimal repros
• Label differences as bug or intended and generate vitest regression tests

Evidence

DiffWitness checks whether two versions of a pure TypeScript function still behave the same. A local open-weight model (Gemma 4 e4b, via Ollama) reads both versions and proposes inputs where they might differ.

Source

The model's guess was "00123". DiffWitness shrank it to the smallest input that still shows the bug (excerpt from the replay run): #2 normalizeOrderId("00") old → "00" new → "0" model args: ("00123") shrunk in 6 steps; why: SILENT DIVERGENCE: OLD preserves leading zeros, but NEW (via parseInt) strips them.

Source

I wrote 22 refactors with one planted silent bug each and adapted 3 real query-string bug fixes. Each case ran 3 times per model (the real fixes on e4b only), plus 2,000 seeded random inputs generated from the declared parameter types.

Source