My Local AI Invented 162 Python Packages. I Reran Everything From Scratch, and 161 Came Back

Description and use

Open‑weight LLMs frequently hallucinate non‑existent Python package names. Using TabPFN and a Claude Code hook, all 212 ghost installs were blocked in 1.7 s, with only 13 real installs mistakenly stopped.

Problem

Local LLMs frequently hallucinate non‑existent Python package names, risking supply‑chain attacks and wasted execution.

Outcome

The guard blocked every ghost install in 1.7 s and only 13 real installs were incorrectly stopped (0.9% false‑positive rate). TabPFN achieved AUROC 0.975 on unseen prompts.

Workflow

• Generate 200 prompts with a fixed template
• Run each prompt twice through each model
• Collect pip install suggestions
• Check each name against PyPI
• Train TabPFN on ghost vs real names
• Deploy phantom‑check CLI and Claude Code hook
• Re‑run study with guard in place
• Measure blocking latency and false positives

Evidence

212 of 1,410 install suggestions (15.0%) named packages that nobody has ever published.

Source

161 of the 162 ghost names came back, and not one new ghost appeared.

Source

TabPFN-3.5 model scores a package name offline. AUROC 0.975 on unseen prompts.

Source

The guard has one job: stand between your agent and pip install, say no to packages that were never written, and let real ones through in 1.7 seconds.

Source

The guard blocked 212 of 212 ghost installs and wrongly stopped 13 of 1,165 real ones.

Source