I built 13 labelled OpenAPI breaking-change cases, and oasdiff missed 2 of them

Description and use

The author created 13 synthetic OpenAPI change cases, 10 of which truly break consumers. Testing oasdiff found 8 of 10, missed 2, and had no false positives, revealing a gap between spec‑level and consumer‑level impact.

Problem

Assessing the accuracy of OpenAPI diff tools in detecting breaking changes that affect consumers.

Outcome

oasdiff identified 8/10 breaking changes, missed 2 that affect consumers, and had no false alarms, highlighting a gap between spec‑level and consumer‑level breaking detection.

Workflow

• Create 13 synthetic OpenAPI spec pairs with labels and consumer scenarios.
• Run oasdiff v1.33.0 using --fail-on ERR on the benchmark set.
• Record which changes are detected, missed, or falsely flagged.
• Analyze gaps between spec‑level and consumer‑level breaking changes.
• Plan to extend to real‑project cases and automated consumer scoring.

Evidence

I built 13 labelled OpenAPI breaking-change cases, and oasdiff missed 2 of them

Source

It caught 8, missed 2, raised no false alarms

Source

The misses: removing an optional response field, and adding an API key requirement to an operation

Source