We scanned 1,939 repositories. 95% of projects publishing an OpenAPI spec have no compatibility gate

Description and use

Most OpenAPI projects lack automated compatibility gates, yet many actively disclose breaking changes. The study corrected the initial 95 % figure to 67 % disclosure rate, highlighting the need for reliable gates and better tooling integration.

Problem

Many OpenAPI‑publishing projects lack automated compatibility checks, risking silent breaking changes for consumers.

Outcome

The study revealed that while most OpenAPI projects lack automated compatibility gates, many teams proactively disclose breaking changes. The initial claim of 95% of projects flying blind was inflated due to ordering, parallel line, and vocabulary errors; the corrected figure shows 67% of breaking changes are disclosed. The data underscores the need for reliable gates and better tooling integration.

Workflow

• Harvest candidate repos via GitHub issue‑search queries for breaking‑change complaints (~2,800 hits).
• De‑noise by requiring ≥2 comments, yielding 1,939 repos.
• Filter for first‑party OpenAPI specs, excluding vendor, node_modules, testdata, etc., leaving 85 repos.
• Check workflow files for known OpenAPI diff tools; only 4 repos used oasdiff.
• For each repo, fetch up to ten consecutive releases, diff adjacent specs, classify changes as consumer‑visible or not.
• Correct for version ordering, parallel maintenance lines, and direction of change to avoid over‑counting.
• Identify and count disclosed breaking changes via release notes, adjusting for vocabulary and length limits.
• Compile results and publish datasets for transparency.
• Share findings and lessons learned with the community.

Evidence

Of 85 repositories with a genuine first‑party OpenAPI spec, exactly four run any breaking‑change detection at all. That is 95.3% with no compatibility gate

Source

Of the 18 repositories that shipped a consumer‑visible breaking change, 12 disclosed every one of them — 67%.

Source

The largest single case was a project that renamed its entire core resource — every /v1/items* route replaced by /v1/entities* — in one release.

Source