How We Tested Railo on 11 Public Repositories

Description and use

Railo, a GitHub Action, found 923 issues in 11 public repositories but proposed 162 fixes, 95 of which were incorrect, mainly in third‑party code and test data. After rule updates, correct fixes dropped to 67.

Problem

Railo proposed many incorrect fixes when run on real public repositories, especially in vendored code, test data, and non‑secret contexts.

Outcome

Railo’s initial run produced 162 fixes, 95 of which were incorrect. After rule adjustments, the number of correct fixes dropped to 67, eliminating noise from vendored code, test data, and non‑secret contexts.

Workflow

• Shallow clone each repository and scan the entire repo instead of a PR diff
• Run Railo’s code path: load config, scan with pinned Semgrep 1.102.0, generate fixes, collect changed files
• Parse each changed file before and after the fix with language‑specific tooling (Python compile, TypeScript parser, gofmt -e, php -l, javac) to ensure no syntax errors
• Read every proposed fix manually
• Identify incorrect fixes and update rules accordingly
• Re‑run harness and read diffs before each Action release

Evidence

923 findings across the eleven repositories.

Source

162 proposed fixes. We read all of them.

Source

95 of those 162 should not have been proposed. Most were edits to vendored libraries, test data and seed data.

Source

None of the 162 broke the syntax of a file.

Source

WebGoat alone accounts for 80 of the 95 removed fixes, and about 77 of those were in two vendored editor files.

Source