Lazy CI Triage and the limits of a correct citation

Description and use

Lazy CI Triage CLI automatically extracts failed test data and classifies it, yet the model’s added value over rules remains unproven.

Problem

Automating the extraction and categorization of failed test information while keeping evidence separate from model interpretation.

Outcome

The tool reliably extracts test metadata and produces validated reports, but model‑added value over rule‑based selection is unproven; both approaches achieve similar validation success.

Workflow

• 1. Run the CLI with a test log file and desired format (e.g., python -m lazy_ci_triage fixtures/lexical-trap.log --format pytest --public-data --mode inventory --output reports/article-inventory).
• 2. The tool parses the log, extracts test identifiers and failure details.
• 3. A local model selects a quote and tentative category based on finite choices.
• 4. A validator checks that the quote matches the record and that the category is supported; otherwise it abstains.
• 5. The report is written, separating Python observations, model selection, selected quote, and uncertainty.
• 6. Users review the report to decide on debugging actions.

Evidence

Lazy CI Triage is a small experimental Python CLI. It reads JUnit XML or supported pytest short summaries and makes an index of failed tests.

Source

We have not measured human time savings or shown that the model adds value over simple rules.

Source

The separate six-record stress test was poor: only 1/5 earlier and 2/5 revised reports passed validation.

Source

The recorded runs used Qwen2.5-Coder 1.5B Instruct Q4_K_M through Ollama 0.18.0, on an RTX 3090 with 24 GiB VRAM and a Ryzen 7 5800X.

Source