Measured on real public CI logs: what two small tools actually did, and where they fell short

说明与用途

在真实的 GitHub Actions 日志上评估两款轻量级 CI 日志分析工具，发现它们都能检测到第一个错误，但未能提供完整的每个测试报告，且有时会错误地归因失败。

问题

Assessing the effectiveness of two lightweight CI log analysis tools on real public logs.

结果

Both tools found the first error in each log, but neither produced complete per-test verdicts. context-diet’s dedupe stage was ineffective on raw logs due to unique timestamps, and its summarization was highly budget‑dependent. ci-failure-detective misattributed failures in Log B and could not generate verdicts without verbose pass records.

工作流

• Clone repositories
• Download raw logs
• Strip timestamps and ANSI codes from Log B
• Run context-diet with dedupe, pack, and extract stages
• Run ci-failure-detective on raw and stripped logs
• Compare tool outputs to ground truth

证据

We replicated the procedure from the tool's own README example: pack with a 400-token budget and the query "test failure".

来源

On Log A, the tool found the first error correctly: line 449 of the downloaded log, FAIL tests/format/js/errors/export-default-from/format.test.js, with the snapshot diff as context.

来源

The tool found the first error correctly: FAIL e2e/tests/customReporters.test.ts (9.769 s), and reported the framework as jest.

来源

The tool could not name the four failed checks.

来源