How We Tested Railo on 11 Public Repositories

Описание и применение

Railo, GitHub Action, обнаружил 923 проблемы в 11 публичных репозиториях, но предложил 162 исправления, из которых 95 были неверными, в основном в сторонних библиотеках и тестовых данных. После обновления правил количество корректных исправлений сократилось до 67.

Задача

Railo proposed many incorrect fixes when run on real public repositories, especially in vendored code, test data, and non‑secret contexts.

Результат

Railo’s initial run produced 162 fixes, 95 of which were incorrect. After rule adjustments, the number of correct fixes dropped to 67, eliminating noise from vendored code, test data, and non‑secret contexts.

Рабочий процесс

• Shallow clone each repository and scan the entire repo instead of a PR diff
• Run Railo’s code path: load config, scan with pinned Semgrep 1.102.0, generate fixes, collect changed files
• Parse each changed file before and after the fix with language‑specific tooling (Python compile, TypeScript parser, gofmt -e, php -l, javac) to ensure no syntax errors
• Read every proposed fix manually
• Identify incorrect fixes and update rules accordingly
• Re‑run harness and read diffs before each Action release

Доказательства

923 findings across the eleven repositories.

Источник

162 proposed fixes. We read all of them.

Источник

95 of those 162 should not have been proposed. Most were edits to vendored libraries, test data and seed data.

Источник

None of the 162 broke the syntax of a file.

Источник

WebGoat alone accounts for 80 of the 95 removed fixes, and about 77 of those were in two vendored editor files.

Источник