How We Tested Railo on 11 Public Repositories

说明与用途

Railo 是一个 GitHub Action，在 11 个公共仓库中发现了 923 个问题，但提出了 162 个修复，其中 95 个是错误的，主要涉及第三方代码和测试数据。更新规则后，正确修复数量降至 67。

问题

Railo proposed many incorrect fixes when run on real public repositories, especially in vendored code, test data, and non‑secret contexts.

结果

Railo’s initial run produced 162 fixes, 95 of which were incorrect. After rule adjustments, the number of correct fixes dropped to 67, eliminating noise from vendored code, test data, and non‑secret contexts.

工作流

• Shallow clone each repository and scan the entire repo instead of a PR diff
• Run Railo’s code path: load config, scan with pinned Semgrep 1.102.0, generate fixes, collect changed files
• Parse each changed file before and after the fix with language‑specific tooling (Python compile, TypeScript parser, gofmt -e, php -l, javac) to ensure no syntax errors
• Read every proposed fix manually
• Identify incorrect fixes and update rules accordingly
• Re‑run harness and read diffs before each Action release

证据

923 findings across the eleven repositories.

来源

162 proposed fixes. We read all of them.

来源

95 of those 162 should not have been proposed. Most were edits to vendored libraries, test data and seed data.

来源

None of the 162 broke the syntax of a file.

来源

WebGoat alone accounts for 80 of the 95 removed fixes, and about 77 of those were in two vendored editor files.

来源