I built 13 labelled OpenAPI breaking-change cases, and oasdiff missed 2 of them

Описание и применение

Автор создал 13 синтетических случаев изменения OpenAPI, из которых 10 действительно ломают потребителей. При тестировании oasdiff обнаружил 8 из 10, пропустил 2, но не выдал ложных тревог, показывая разрыв между спецификацией и реальным влиянием на потребителей.

Задача

Assessing the accuracy of OpenAPI diff tools in detecting breaking changes that affect consumers.

Результат

oasdiff identified 8/10 breaking changes, missed 2 that affect consumers, and had no false alarms, highlighting a gap between spec‑level and consumer‑level breaking detection.

Рабочий процесс

• Create 13 synthetic OpenAPI spec pairs with labels and consumer scenarios.
• Run oasdiff v1.33.0 using --fail-on ERR on the benchmark set.
• Record which changes are detected, missed, or falsely flagged.
• Analyze gaps between spec‑level and consumer‑level breaking changes.
• Plan to extend to real‑project cases and automated consumer scoring.

Доказательства

I built 13 labelled OpenAPI breaking-change cases, and oasdiff missed 2 of them

Источник

It caught 8, missed 2, raised no false alarms

Источник

The misses: removing an optional response field, and adding an API key requirement to an operation

Источник