The Registry That Wasn't Enough: How We Went from "Knowing APIs" to Actually Verifying Against PyPI

Description and use

A workflow that detects hallucinated imports and API calls in AI‑generated code by automatically extracting package contents from PyPI wheels, employing caching and a deep‑analysis mode to reduce false positives and improve recall.

Problem

Detecting hallucinated library imports and API calls in AI‑generated code without relying on a handcrafted registry.

Outcome

Second run on same file dropped from ~50s to 17ms; false positive for pandas.DataFrame eliminated; new false positives for nonexistent APIs detected; precision on clean code remains perfect.

Workflow

• 1. Parse import statements in the target .py file.
• 2. For each imported package, check cache for existence and metadata (TTL 7/30 days).
• 3. If not cached, query PyPI; on success, store metadata and wheel URL.
• 4. Canonicalize package names (e.g., PIL→Pillow).
• 5. If deep mode enabled and within IP limit, download wheel (cached per version).
• 6. Open wheel ZIP, search .pyi and .py files for referenced attributes using regex.
• 7. Classify each reference as confirmed, not found, or unverifiable.
• 8. Aggregate findings, report precision/recall, and provide evidence snippets.
• 9. Respect fail‑open: on network errors or timeouts, skip verification rather than flag as nonexistent.

Evidence

The tool I built back then relied on a handcrafted registry of valid APIs: a map of library → known functions, and a simple rule: if the library is in the registry and the function isn't, it's a hallucination.

Source

PyPI doesn't just serve packages: it serves wheels, which are ZIP files containing the code and — very importantly — the type stubs (.pyi) when the package includes them.

Source

The result is noticeable: the second run on the same file goes from ~50 seconds to 17 milliseconds.

Source

Now there's a deep mode (deep) that, when you enable it: Queries PyPI for the package. Downloads the wheel (once, cached per version). Opens the ZIP and searches for the attribute in all the package's .pyi and .py files.

Source