Gemma Hallucinated 29 npm/PyPI Packages. I Built a Tool to Catch Them.

Description and use

Gemma hallucinated 29 non‑existent npm/PyPI packages, risking unsafe installations. The ghostdeps tool checks AI‑suggested package names against registries, classifying them to prevent installation of ghost dependencies.

Problem

AI language models can hallucinate non‑existent package names, risking accidental installation of malicious or non‑existent code.

Outcome

ghostdeps identified 29 out of 118 (24.6%) hallucinated packages in 535 Gemma prompts, with nearly half of niche tasks producing ghosts. Re‑runs showed high repeatability of hallucinations, indicating a systemic issue.

Workflow

• 1. Prompt the local AI model for a package recommendation. 2. Parse the model’s answer to extract install commands. 3. Query npm and PyPI registries for each extracted package name. 4. Classify each name as alive, GHOST, POSSESSED, dead, tombstone, EXORCISED, or unknown. 5. Report the classification results to the user.
• 6. Optionally rerun the prompt with different seeds or higher temperature to test repeatability of hallucinations.
• 7. Use ghostdeps check on a package.json, requirements file, or pasted chatbot answer to pre‑validate dependencies before installation.

Evidence

The npm registry has never heard of that package. The GitHub repository returns a 404.

Source

ghostdeps is a tool for detecting AI-hallucinated dependencies before they make it into your project.

Source

535 questions, all local, on a laptop with a 4 GB GPU, about 70 minutes of model time.

Source

29 of 118 (24.6%)

Source