Count the Argument Shapes Your Fixtures Never See Before Refactoring a Messy Module

Description and use

A script mines call sites, classifies argument shapes, and generates characterization tests for unseen shapes, preventing bugs after refactoring.

Problem

A refactor can introduce bugs when test suites never exercise certain input shapes, leading to failures that are not caught by coverage tools.

Outcome

The workflow ensures that refactors do not introduce bugs caused by untested input shapes, bridging the gap between coverage metrics and real input diversity.

Workflow

• 1. Run the call_shapes.py script to collect all call sites of the target function and classify each argument into coarse shape buckets.
• 2. Compare the collected shape buckets with those exercised by existing tests.
• 3. For any shape not covered by tests, create a new characterization test row.
• 4. Refactor the module only after all unseen shapes have corresponding tests.

Evidence

Three days later a batch job that passes an empty line-item list starts returning None instead of 0.0, because no test ever called that function with an empty collection.

Source

Coverage tools answer a different question than the one you are asking. A line-coverage report tells you which statements executed, but it says nothing about the diversity of values that reached them.

Source

The following script is plain Python with no dependencies beyond the standard library, and it works on any repository that parses as valid Python.

Source