I Imported the Patch. That's When It Died.

说明与用途

三步验证（解析、导入、测试）捕获了仅靠 ast.parse 无法发现的导入时错误，确保仅通过完整功能的补丁。

问题

Automated evaluation of code fixes often relies on superficial checks like parsing or syntax validation, missing deeper runtime failures such as import-time errors or missing dependencies.

结果

The harness successfully identified import‑time failures that ast.parse alone missed, ensuring that only fully functional patches passed the evaluation.

工作流

• 1. Define a minimal fixture module that imports a missing dependency and raises at import time.
• 2. Write a test that imports the module and asserts correct functionality.
• 3. Implement a parse gate that runs ast.parse on the source file.
• 4. Implement an import gate that imports the module in a fresh process to catch import‑time errors.
• 5. Implement a test gate that runs the test suite and verifies assertions.
• 6. Extract the unified diff from the model’s completion using a regex extractor.
• 7. Apply the diff with the Unix patch tool in a temporary directory.
• 8. Run the three gates sequentially, scoring each gate independently.
• 9. Aggregate scores to determine overall fix quality.

证据

ast.parse will not run this. require_tax_table()

来源

If the completion is a memoir with a fenced snippet in the middle, and that snippet is not a unified diff, the extractor returns None.

来源