I Imported the Patch. That's When It Died.

Описание и применение

Автоматизированный набор из трёх проверок (парсинг, импорт, тест) выявил ошибки, которые пропускали простые синтаксические проверки, гарантируя корректность импортируемого модуля.

Задача

Automated evaluation of code fixes often relies on superficial checks like parsing or syntax validation, missing deeper runtime failures such as import-time errors or missing dependencies.

Результат

The harness successfully identified import‑time failures that ast.parse alone missed, ensuring that only fully functional patches passed the evaluation.

Рабочий процесс

• 1. Define a minimal fixture module that imports a missing dependency and raises at import time.
• 2. Write a test that imports the module and asserts correct functionality.
• 3. Implement a parse gate that runs ast.parse on the source file.
• 4. Implement an import gate that imports the module in a fresh process to catch import‑time errors.
• 5. Implement a test gate that runs the test suite and verifies assertions.
• 6. Extract the unified diff from the model’s completion using a regex extractor.
• 7. Apply the diff with the Unix patch tool in a temporary directory.
• 8. Run the three gates sequentially, scoring each gate independently.
• 9. Aggregate scores to determine overall fix quality.

Доказательства

ast.parse will not run this. require_tax_table()

Источник

If the completion is a memoir with a fenced snippet in the middle, and that snippet is not a unified diff, the extractor returns None.

Источник