My tool called a maintainer's plugin broken. They had fixed it two weeks earlier.

Description and use

A nightly job installs community-maintained plugins from Pipecat docs into a fresh environment, imports them, and checks that every service class can still be constructed. The checker labels packages as 'broken', but this conflates two questions: whether users get a broken package, and whether the maintainer's code is broken. The fix probes the plugin's own source repository when the released version fails. Of 13 failing plugins, 8 are still broken from source, 1 is fixed in source but stale on PyPI, and 4 are unreadable. Zero vendors have fixed their own plugin.

Problem

A nightly job installs community-maintained plugins from Pipecat docs into a fresh environment, imports them, and checks that every service class can still be constructed. The checker labels packages as 'broken', but the label conflates two different questions: whether users are getting a package that doesn't work, and whether the maintainer's code is broken.

Outcome

Of 13 currently failing plugins, 8 are still broken when built from source, 1 is fixed in source but stale on PyPI, and 4 could not be read (reported unknown). Zero of the nine vendors has fixed their own plugin. The checker now distinguishes 'users get a broken package' from 'code is broken' by probing the plugin's own source repository.

Workflow

• Install every community-maintained plugin listed in Pipecat docs into a fresh environment
• Import each plugin and check that every service class it defines can still be constructed
• Publish a table: works, broken, pinned to an old version, can't install

Evidence

It installs from PyPI, because that's what a user gets.

That landed on 30 September. PyPI still serves the version published before it. So for more than two weeks my dashboard printed a red row, with their project's name on it, for code that was already correct.

There are two different sentences hiding in one red row: 'Your users are getting a package that doesn't work.' True for Boson. Still true today. 'Your code is broken.' False for Boson, and false for two weeks.

pipecat-boson declares nothing. No Homepage, no Repository, no project_urls at all. Its repository appears in exactly one place in the entire package metadata: a git clone line in the README body.

All four published URLs are a template placeholder that was never filled in. Anyone who installs that package and hits a problem follows the Bug Tracker link to a 404.

pipecat-supertonic resolves to supertone-inc/supertonic — the vendor's own text-to-speech model repository, thirteen thousand stars — because the two names share a word.

Measuring something in public means you can be wrong about it in public. The guards are the work, not the measurement.

One. Not a stack. And the harder number: zero of the nine vendors has fixed their own plugin.