My Local AI Invented 162 Python Packages. I Reran Everything From Scratch, and 161 Came Back

说明与用途

开源 LLM 经常生成不存在的 Python 包名。通过 TabPFN 和 Claude Code 钩子，所有 212 个“幽灵”安装在 1.7 秒内被阻止，误阻 13 个真实安装。

问题

Local LLMs frequently hallucinate non‑existent Python package names, risking supply‑chain attacks and wasted execution.

结果

The guard blocked every ghost install in 1.7 s and only 13 real installs were incorrectly stopped (0.9% false‑positive rate). TabPFN achieved AUROC 0.975 on unseen prompts.

工作流

• Generate 200 prompts with a fixed template
• Run each prompt twice through each model
• Collect pip install suggestions
• Check each name against PyPI
• Train TabPFN on ghost vs real names
• Deploy phantom‑check CLI and Claude Code hook
• Re‑run study with guard in place
• Measure blocking latency and false positives

证据

212 of 1,410 install suggestions (15.0%) named packages that nobody has ever published.

来源

161 of the 162 ghost names came back, and not one new ghost appeared.

来源

TabPFN-3.5 model scores a package name offline. AUROC 0.975 on unseen prompts.

来源

The guard has one job: stand between your agent and pip install, say no to packages that were never written, and let real ones through in 1.7 seconds.

来源

The guard blocked 212 of 212 ghost installs and wrongly stopped 13 of 1,165 real ones.

来源