Measured on real public CI logs: what two small tools actually did, and where they fell short

Description and use

Evaluating two lightweight CI log analysis tools on real public GitHub Actions logs revealed that both detected the first error but failed to provide complete per-test reports and sometimes misattributed failures.

Problem

Assessing the effectiveness of two lightweight CI log analysis tools on real public logs.

Outcome

Both tools found the first error in each log, but neither produced complete per-test verdicts. context-diet’s dedupe stage was ineffective on raw logs due to unique timestamps, and its summarization was highly budget‑dependent. ci-failure-detective misattributed failures in Log B and could not generate verdicts without verbose pass records.

Workflow

• Clone repositories
• Download raw logs
• Strip timestamps and ANSI codes from Log B
• Run context-diet with dedupe, pack, and extract stages
• Run ci-failure-detective on raw and stripped logs
• Compare tool outputs to ground truth

Evidence

We replicated the procedure from the tool's own README example: pack with a 400-token budget and the query "test failure".

Source

On Log A, the tool found the first error correctly: line 449 of the downloaded log, FAIL tests/format/js/errors/export-default-from/format.test.js, with the snapshot diff as context.

Source

The tool found the first error correctly: FAIL e2e/tests/customReporters.test.ts (9.769 s), and reported the framework as jest.

Source

The tool could not name the four failed checks.

Source