Measured on real public CI logs: what two small tools actually did, and where they fell short

Описание и применение

Оценка двух инструментов анализа логов CI на реальных публичных логах GitHub Actions показала, что оба обнаружили первую ошибку, но не смогли предоставить полные отчёты о тестах и иногда ошибочно определяли местоположение ошибок.

Задача

Assessing the effectiveness of two lightweight CI log analysis tools on real public logs.

Результат

Both tools found the first error in each log, but neither produced complete per-test verdicts. context-diet’s dedupe stage was ineffective on raw logs due to unique timestamps, and its summarization was highly budget‑dependent. ci-failure-detective misattributed failures in Log B and could not generate verdicts without verbose pass records.

Рабочий процесс

• Clone repositories
• Download raw logs
• Strip timestamps and ANSI codes from Log B
• Run context-diet with dedupe, pack, and extract stages
• Run ci-failure-detective on raw and stripped logs
• Compare tool outputs to ground truth

Доказательства

We replicated the procedure from the tool's own README example: pack with a 400-token budget and the query "test failure".

Источник

On Log A, the tool found the first error correctly: line 449 of the downloaded log, FAIL tests/format/js/errors/export-default-from/format.test.js, with the snapshot diff as context.

Источник

The tool found the first error correctly: FAIL e2e/tests/customReporters.test.ts (9.769 s), and reported the framework as jest.

Источник

The tool could not name the four failed checks.

Источник