auditsift: security-audit every PR with an AI agent, without drowning in false positives

Описание и применение

auditsift позволяет проверять PR‑ы на наличие уязвимостей, используя сначала офлайн‑сканирование, а затем два этапа LLM‑проверки, требуя подтверждения кода, что снижает ложные срабатывания и расходы.

Задача

AI‑based security reviews of PRs often return noisy, untrustworthy findings and incur high costs when run on every pull request.

Результат

The workflow reduces false positives by requiring quoted evidence, limits LLM usage to a few snippets, and keeps most checks offline, enabling free audits with optional paid explanations.

Рабочий процесс

• Run offline "map" phase to detect languages, entry points, routes, dependencies and redact secrets.
• Run offline "scan" phase applying hardcoded rules for secrets, SQL interpolation, shell usage, eval, pickle, yaml.load, weak hashes, verify=False, path traversal, SSRF, permissive CORS.
• Send only candidate snippets to a triage LLM for impact and fix suggestions.
• Send the same snippets to a second, independent verify LLM that must quote existing code before confirming a finding.
• Generate JSON, SARIF 2.1.0, and Markdown reports; use "--fail-on high" to block CI if high‑severity findings remain.
• Filter findings to the PR diff with "--diff" and run the audit in a GitHub Actions job that uploads SARIF to the Security tab.
• Optionally enable LLM passes by setting API keys and model names; use AtmoRouter for cost‑effective access.
• Integrate with agents via "auditsift serve" MCP server to call map/scan/audit only when "use_llm:true".

Доказательства

Ask a coding agent to "review this PR for security issues" and you usually get back a long, confident list. Some of it is real. A lot of it is plausible-sounding noise

Источник

auditsift runs an audit in phases. The first two need no API key and never touch the network.

Источник

The verify step is the part I care about most. It doesn't see the triage write-up, so it can't just agree with it. A "confirmed" verdict only counts when the quoted evidence is found in the code.

Источник

Exit code 1 means a kept finding is at or above the gate, so CI blocks the merge.

Источник