Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

The Registry That Wasn't Enough: How We Went from "Knowing APIs" to Actually Verifying Against PyPI

说明与用途

一种通过从 PyPI wheel 中自动提取包内容、使用缓存和深度分析模式来检测 AI 生成代码中的“幻觉”导入和 API 调用的工作流,降低误报并提高召回率。

问题

Detecting hallucinated library imports and API calls in AI‑generated code without relying on a handcrafted registry.

结果

Second run on same file dropped from ~50s to 17ms; false positive for pandas.DataFrame eliminated; new false positives for nonexistent APIs detected; precision on clean code remains perfect.

工作流

  • 1. Parse import statements in the target .py file.
  • 2. For each imported package, check cache for existence and metadata (TTL 7/30 days).
  • 3. If not cached, query PyPI; on success, store metadata and wheel URL.
  • 4. Canonicalize package names (e.g., PIL→Pillow).
  • 5. If deep mode enabled and within IP limit, download wheel (cached per version).
  • 6. Open wheel ZIP, search .pyi and .py files for referenced attributes using regex.
  • 7. Classify each reference as confirmed, not found, or unverifiable.
  • 8. Aggregate findings, report precision/recall, and provide evidence snippets.
  • 9. Respect fail‑open: on network errors or timeouts, skip verification rather than flag as nonexistent.

证据

The tool I built back then relied on a handcrafted registry of valid APIs: a map of library → known functions, and a simple rule: if the library is in the registry and the function isn't, it's a hallucination.

来源

PyPI doesn't just serve packages: it serves wheels, which are ZIP files containing the code and — very importantly — the type stubs (.pyi) when the package includes them.

来源

The result is noticeable: the second run on the same file goes from ~50 seconds to 17 milliseconds.

来源

Now there's a deep mode (deep) that, when you enable it: Queries PyPI for the package. Downloads the wheel (once, cached per version). Opens the ZIP and searches for the attribute in all the package's .pyi and .py files.

来源

  • 记录日期: 2026-11-03T20:14:16.295Z