I built 13 labelled OpenAPI breaking-change cases, and oasdiff missed 2 of them

说明与用途

作者创建了13个合成的OpenAPI更改案例，其中10个真正导致消费者破坏。使用oasdiff测试发现8个，漏掉2个，且没有误报，显示了规范级别与消费者级别影响之间的差距。

问题

Assessing the accuracy of OpenAPI diff tools in detecting breaking changes that affect consumers.

结果

oasdiff identified 8/10 breaking changes, missed 2 that affect consumers, and had no false alarms, highlighting a gap between spec‑level and consumer‑level breaking detection.

工作流

• Create 13 synthetic OpenAPI spec pairs with labels and consumer scenarios.
• Run oasdiff v1.33.0 using --fail-on ERR on the benchmark set.
• Record which changes are detected, missed, or falsely flagged.
• Analyze gaps between spec‑level and consumer‑level breaking changes.
• Plan to extend to real‑project cases and automated consumer scoring.

证据

I built 13 labelled OpenAPI breaking-change cases, and oasdiff missed 2 of them

来源

It caught 8, missed 2, raised no false alarms

来源

The misses: removing an optional response field, and adding an API key requirement to an operation

来源