I let a small local model write the refactor tests I was too lazy to write

说明与用途

DiffWitness 使用本地 Gemma 4 模型检测重构后隐藏的回归错误，并自动生成失败的回归测试。

问题

Refactor changes can introduce silent bugs that existing tests miss, especially edge cases like leading zeros in order IDs.

结果

DiffWitness identified a silent bug caused by the refactor, automatically generated a regression test, and demonstrated that a small local LLM can effectively find edge‑case regressions.

工作流

• Load old and new function versions via TypeScript compiler API
• Instrument branches with counters
• Prompt Gemma 4 e4b to propose up to eight input values
• Validate and filter proposed inputs against type constraints
• Execute both versions in isolated VM contexts and compare results
• Shrink any detected differences to minimal repros
• Label differences as bug or intended and generate vitest regression tests

证据

DiffWitness checks whether two versions of a pure TypeScript function still behave the same. A local open-weight model (Gemma 4 e4b, via Ollama) reads both versions and proposes inputs where they might differ.

来源

The model's guess was "00123". DiffWitness shrank it to the smallest input that still shows the bug (excerpt from the replay run): #2 normalizeOrderId("00") old → "00" new → "0" model args: ("00123") shrunk in 6 steps; why: SILENT DIVERGENCE: OLD preserves leading zeros, but NEW (via parseInt) strips them.

来源

I wrote 22 refactors with one planted silent bug each and adapted 3 real query-string bug fixes. Each case ran 3 times per model (the real fixes on e4b only), plus 2,000 seeded random inputs generated from the declared parameter types.

来源