My Local AI Invented 162 Python Packages. I Reran Everything From Scratch, and 161 Came Back

Описание и применение

Модели с открытым весом часто генерируют несуществующие имена пакетов Python, что может привести к атакам. С помощью TabPFN и хука Claude Code удалось заблокировать все 212 «призрачных» установок за 1,7 с, при этом ошибочно заблокировано только 13 реальных.

Задача

Local LLMs frequently hallucinate non‑existent Python package names, risking supply‑chain attacks and wasted execution.

Результат

The guard blocked every ghost install in 1.7 s and only 13 real installs were incorrectly stopped (0.9% false‑positive rate). TabPFN achieved AUROC 0.975 on unseen prompts.

Рабочий процесс

• Generate 200 prompts with a fixed template
• Run each prompt twice through each model
• Collect pip install suggestions
• Check each name against PyPI
• Train TabPFN on ghost vs real names
• Deploy phantom‑check CLI and Claude Code hook
• Re‑run study with guard in place
• Measure blocking latency and false positives

Доказательства

212 of 1,410 install suggestions (15.0%) named packages that nobody has ever published.

Источник

161 of the 162 ghost names came back, and not one new ghost appeared.

Источник

TabPFN-3.5 model scores a package name offline. AUROC 0.975 on unseen prompts.

Источник

The guard has one job: stand between your agent and pip install, say no to packages that were never written, and let real ones through in 1.7 seconds.

Источник

The guard blocked 212 of 212 ghost installs and wrongly stopped 13 of 1,165 real ones.

Источник